These engines are described as language models. That is an appropriate term, but we need to think about what we mean by “language”. Words in a language don’t have intrinsic meaning. That meaning comes from context. The word you say and the word I hear may be different. Alas, the very language we use to talk about this is hobbled by the same ambiguities.

Take the word “hear” – it can mean to simply detect the soundwaves, as in a typical hearing test. It can also mean hearing in the sense of understanding, as in hearing what you’re saying. An interesting example is the difference between a pause and a pregnant pause. An observer doesn’t even know that meaning is being conveyed, let alone whether the listener (is that the appropriate word) is anxiously awaiting the end of the silence.

A key point is that the relationship between the endpoints of the conversation is independent of whatever is between them. And in the case of silence, there may be nothing.

I want to keep this essay short, so I won’t go into the ability of those along the path to distort or block the exchange or to listen in. What meddlers can’t do is know for sure what is happening in people’s heads though they might guess or have a theory of mind.

The idea that we are communicating between two endpoints or even within one endpoint (as in thinking to ourselves) by the exchange of symbols is the same concept that has made the Internet so transformative and so fundamentally different from traditional telecommunications, which purported to transport meaning.

As I wrote in the Age of Software, the idea of decoupling meaning from symbols, bits as representation, is powerful and fundamental. The idea that we can use a common packet infrastructure for all exchanges is an idea that is still difficult for people to fully appreciate. Hence, we persist in public policies, oblivious to this new landscape.

It's also why I’ve compared the conceptual shift to the shift from geocentrism to heliocentrism. We were no longer at the center of the universe, and now, meaning is no longer intrinsic. There is no intrinsic good or evil. So much of our public policy and social constructs are based on the presumption of intrinsic meaning.

It's telling that so much of our view of these AI engines has been informed by looking at how it writes prose or generates images which are soft targets which makes it easy to miss the limitations, though, in Why AI art struggles with hands, you can get a good sense of the limitations. There’s a reason that DaVinci did dissections – he needed to find the hidden mechanisms and not just guess from the surface manifestations.

I use the tools of a programmer assistant, which brings these limitations into sharp relief. It’s like working with an enthusiastic assistant who knows tropes but doesn’t really understand what is going on. In a similar vein, when I do ask for an essay about a technical topic, I see an echo of conventional wisdom that resists innovative thinking. Kind of ironic.

For that matter, such assistance is very frustrating when I’m writing – I use the tools to catch many errors, but I spend just as much time fighting back as it fights any attempt to use modern concepts where words have a different nuance, if not meaning.

Alas, as Raskin points out, these limitations aren’t an issue when we use the technology in service of misinformation. That’s a deep issue in its own right.

Stepping back, there is a deep philosophical issue about where meaning comes from. One way I think of it is as patterns. Heliocentrism is a good example. Ptolemy and Copernicus were observers seeing patterns. There was no difference in the underlying reality but all the difference in the world (and out of it) in how they saw it.

The approach of finding patterns is fundamental to how systems evolve. More to the point, coevolve in multiple contexts – there is no simple measure of fitness. The observer is not anthropomorphic, but rather we just use such terminology because that’s also how language works – repurposing with new meanings. And it’s also the problem with language as the meaning of the words change and drift apart. It’s amazing we can communicate at all! And often we fail to even when using the same words!

At this point, we need an articulate understanding of the new concepts in which meaning is not intrinsic and is fundamentally ambiguous (in multiple contexts).

I tried to simplify the model by ignoring distortions in the “between”. That’s fine for a model, but when we take these models further, issues like relative timing come to the fore as another kind of context. And properties we attribute to intent or intelligence might be implicit. We don’t need to learn to walk if our bodies are built to walk – we just tweak the mechanism.

This is why I worry about those who think we can just represent our minds as an array of bits. It’s reminiscent of the early days of mechanical systems and the idea that we could think of brains as simple machines.

The other related idea is the assumption that we can predict the future and make wise choices. Perhaps I’m overly influenced by time in the world of finance in which generations of mavens have hit upon the way to predict the future – until, as Nassim Taleb noted in The Black Swan, they couldn’t. One reason is that if everyone can predict the future, then no one can. (Similar to the halting problem in computing).

The other reason is that all these mechanisms rely on a useful heuristic of locality. This is what has enabled us to share computers when each program seemed to need the entire memory of the computer. We didn’t normally reference the computer’s memory hither and yon, but rather, references stayed relatively local.

Tools like neural nets had a similar problem – being able to extrapolate from local samples. It’s a great heuristic, but it can easily fail, as when people know the earth is flat because that’s all they see.

The value of heliocentrism wasn’t just that it made navigation easier. It gave us insights into the universe at large by finding commonalities among mechanisms. I have the same approach to learning – I don’t feel I’ve learned something until I have an insightful understanding and don’t have to memorize the details. If I know the subject, I don’t need to study – I can figure it out on the test. If I have to memorize, it means I’m not being taught a pattern. Alas, sometimes, as in language, there may not be a simple pattern, and in the AI engines, the seeming correlations might be spurious.