Here’s a recent thread on philosophy of AI from Twitter/X, in which I address rather popular arguments made by Emily Bender and others to the effect that LLM outputs are strictly speaking meaningless. I think these argument are flawed, as I explain below. But I think it’s worth categorising these as post-Searlean critiques, following John Searle’s infamous arguments against the possibility of computational intentionality. I think post-Searlean and post-Dreyfusian critiques form the main strands of contemporary opposition to the possibility of AI technologies developing human-like capacities.

I think it’s maybe worth summarising why I think post-Searlean critics of AI such as Emily Bender are wrong to dismiss the outputs of LLMs as meaningless. Though it’s perhaps best to begin with a basic discussion of what’s at stake in these debates.

Much like the term ‘consciousness’, the term ‘meaning’ often plays proxy for other ideas. For example, saying systems can’t be conscious is often a way of saying they’ll never display certain degrees of intelligence or agency, without addressing the underlying capacities.

Similarly, being able to simply say ‘but they don’t even know what they’re saying’ is a simple way to foreclose further debate about the communicative and reasoning capacities of LLMs without having to pick apart the lower level processes underpinning communication and reasoning.

Now, I’m not going to argue that LLMs in fact know what they are saying, but simply pick apart what I see as over-strong arguments that preclude the possibility that full-blown linguistic competence might be built out of technologies that borrow from these systems.

(If you want a summary of the debate over Bender’s arguments from the linguistics side, see this blog post: https://julianmichael.org/blog/2020/07/23/to-dissect-an-octopus.html…)

So, I think there are two main arguments here: 1) that no matter how much LLMs emulate the ‘form’ of language by encoding structural relations between symbols, their outputs lack content because these symbols aren’t ‘grounded’ by practical relations to the things they represent.

And 2) that no matter how much LLMs emulate the ‘form’ of language by mirroring coherent dialogical interaction, their outputs lack content because the system’s are incapable of the communicative intent underpinning genuine speech acts such as assertion.

I think (1) is more or less cope at this point. There’s not nothing to it, but since the advent of word2vec there’s been a steady stream of practical vindications for structuralism (even if the remaining structuralists in the humanities have largely ignored them).

(A recent example of such a vindication can be found here: https://arxiv.org/abs/2503.21073)

Now, I’m an inferentialist (which, per Jaroslav Peregrin, is a type of structuralism). I think the core component of meaning is given by the role sentences play in ‘the game of giving and asking for reasons’ and the contributions component words make to these inferential roles.

LLMs are not designed to distil inferential relations between words and concatenated sequences thereof, but they do capture them in the course of distilling relations of probabilistic proximity. They also capture the semiotic excess of loose connotation over strict implication.

Inferentialism does have a few things to say about word-world relations: 1) GOGAR extends beyond pure inference to include non-inferential language-entry moves (perception) and language-exit moves (action); this is how expressions get specifically empirical or evaluative content.

And 2) the role of the representational vocabulary we use to make word-world relations explicit (‘of’, ‘about’, etc.) is to enable us to triangulate between divergent perspectives on the implications of our linguistically articulated commitments.

LLMs obviously lack the capacity for perception or action. It is this lack that Bender argues precludes their symbols from being grounded. However, there are many symbols we use that simply aren’t grounded in this way, most importantly mathematical terms and unobservables.

The semantic content of mathematical statements is more or less purely inferential, lacking any direct connection to perception or action. Some (e.g., Derrida) have used this to argue that mathematics is an inherently meaningless game of symbol manipulation, but this is silly.

Mathematics is superlatively capable of investigating its own content, as the history of formal logic and semantics demonstrates. To deny mathematics inherent meaning is to falsify it.

Unobservables (e.g., ‘electron’, ‘black hole’, ‘allele’, etc.) are more significant however. These are terms for entities we can neither directly perceive or act upon, but engage with through experimental and technological practices mediated by complex inferences.

Advocates of the importance of grounding tend to argue that these terms are grounded indirectly by their relations to terms that are grounded directly in experience, such that we couldn’t have meaningful terms for unobservables without some meaningful terms for observables.

Fair enough. But the question is precisely how these word-world connections are inherited. Must it be intra-personal, or can it be inter-personal? Need a physicist be able to describe and calibrate all the instruments in the chain that leads to his black hole data to discuss it?

It seems reasonable to say that two students of physics can have a meaningful discussion of black holes without yet knowing much about the elaborate processes through which they are detected. Indeed, even two lay people can probably have a simple chat about them.

But then, it equally seems that they might have a simple chat about things they haven’t seen, but which aren’t strictly unobservable (e.g., Siberian tigers, the NYSE, bowel cancer). What constrains them here is not their own capacities for perception/action, but those of others.

Meaning is articulated not just by relations of inference, but also relations of deference. Lay people defer to physicists when it comes to black holes. The blind defer to the sighted when it comes to the behaviour of light and pigment. Our talk is grounded by social constraint.

We might want to say that some people have richer understanding of words than others, either because they have a more profound grasp of their inferential complexities or because they have direct practical experience of their referents. But there’s legitimate gradation here.

We thus might think it is feasible for a computer system that lacks perceptual and practical capacities of any kind outside of its communicative channels to feasibly be constrained (and its output thereby grounded) by its relations to the social community whose language it uses.

The question is whether relations between LLMs and the communities whose linguistic behaviour they compress is sufficient to really sustain this. There are some legitimate reasons to think it isn’t. And this brings us back to representational vocabulary.

There’s an emerging consensus that LLMs can develop something like discrete ‘representations’ of certain features of the world in order to predictively generate text ‘about’ them. Yet there’s also a general skepticism that these compose anything like a genuine ‘world model’.

Working backwards from world models, the real challenge is that LLMs are not really interested in truth. This is partly to do with what get’s colloquially called ‘hallucination’ – the tendency to fabulate plausible but false statements. In other words, to misrepresent the world.

But perhaps more important is the problem of consistency. The ‘attention’ windows deployed by LLMs are very large, and enable them to remain mostly self-consistent within a given conversation, but if you push them too far they can quite easily contradict themselves.

More significantly, they will, with no to little prompting, express conflicting opinions across different conversations. They excel at maintaining a range of locally cohesive dialogical interactions, but underlying these there is no globally coherent worldview.

There are in-context hacks that LLM proprietors use to mitigate these problems to some extent. They can secretly frontload our interactions with lists of facts that discourage hallucination about the relevant topics, producing a very crude ersatz world-model.

But not only do these only stabilise outputs on a tiny of fraction of topics the LLM can address, the system cannot update them in response to its interactions with users, even if they may be convinced to ignore them in the context of the current conversation.

And here is the nub of the difference between the social constraint that the community of human language users exercise upon one another and that we exercise on LLMs trained on our behaviour. We aren’t just pre-trained, we continue to train one another through interaction.

And crucially, when these interactions turn upon conflicts of opinion, we can effectively calibrate the way that we involve the world in resolving them. We can establish that we’re talking *about* the same things, even if we disagree about the implications of what we’re saying.

The obvious response to this is that much if not most of the actual social pressure that shapes linguistic usage simply doesn’t involve this sort of explicit clash and rational navigation of competing worldviews in which we chase down the threads that tie our words to the world.

If mostly what we do is just tacitly correct one another’s speech, is this distributed network of behavioural nudges any more significantly constrained by the world itself than the process of distilling behaviour wholesale and then finessing it with RLHF and in-context tweaks?

I think that whatever significance we ultimately grant to explicit disagreement, we should probably recognise that these two forms of implicit constraint (social pressure/ML) are not fundamentally different in kind, even if in degree, and even if one is parasitic on the other.

This brings us to Bender’s second argument regarding the absence of communicative intention, which can be usefully approached from a similar direction. I’ll accept the premise that LLMs don’t possess communicative intentions, but aim to show this is less relevant than it appears.

To unpack the argument, the idea is that when interpreting what an LLM means by what it says (e.g., given ambiguities or unstated implications), it makes no sense to assume there’s a correct way to decide between options, as it simply cannot intend us to pick one.

In other words, even if LLM outputs display some form of literal meaning (contra argument 1) they cannot display any form of speaker meaning. More generally, we cannot interpret the force of outputs as making specific sorts of speech acts (assertion, question, request, etc.).

I think the best counter example to this argument is also pretty illustrative of how LLMs fit into the existing linguistic community: rumours. These are something like free-floating assertions disconnected from their origin, and possibly without one given whisper mutations.

To pass on a rumour is not exactly to make an assertion, but it doesn’t necessarily erase any potential epistemic valence or preclude the possibility of interpretation. We can even still interpret the intentions of the phantom speaker, even if they never really existed.

If you want a comparable example of such seemingly untethered interpretation, consider the energy still put into interpreting (and aiming to correctly translate) figures like Homer.

The output of an LLM is a lot like a rumour: a statement/text produced by a distributed social process in a way that gives it obvious semantic content but severs the deferential links that articulate the social process of justification.

I’ve elsewhere likened LLMs to rumour mills. It’s almost as if we’ve learned to gossip with the distributed social process directly, without passing through its nodes. A fully automated rumour mill we can even inject suggestions into and see them come back fully fleshed out.

Of course, an LLM is much less like an evolving group dynamic than a snapshot of an egregore, and far more tightly focused in the way it develops prompts. There’s emergent novelty where new patterns or capacities arise from the sheer amount of behaviour they compress. But still.

Just as we could argue that the way LLMs’ outputs might be grounded (however poorly) by their connection to the implicit process of distributed social constraint, we can argue that they display something like speaker intention (however poorly) in virtue of the same connection.

Though we may justifiably claim that no random string of shapes worn in the sand by waves can mean anything no matter how much they might look like letters, we must admit that the probabilistic processes underpinning LLMs are closer to speech than they are to waves on the sand.

Now, I’m not claiming these systems literally have communicative intentions, but rather that the way they’re embedded in a socio-linguistic context makes them as it were ‘good enough for government work’. We can disambiguate LLM outputs in ways we can’t disambiguate the sand.

I should add that our ability to dialogue with LLMs certainly seems to help in this regard, as we can ask them to clarify what they mean. There are legitimate questions as to what extent such clarifications are post-hoc fabulations, but such questions often apply to humans too.

This has been a long thread, and I should try to wrap it up, so let me try and weave my different concerns together. If I agree with Bender that LLMs are not full blooded agents, lacking determinate beliefs and intentions, why do I think it is important to dispute her arguments?

A pervasive problem in AI debates is setting the bar for what counts as mindedness. One sees this in AGI talk, where this can mean either task flexibility somewhat less than an average human, or problem solving capacity greater than human society, depending who is talking.

I worry something similar is going on with meaning. Some want to claim that LLMs already fully grasp it because they generate surprisingly coherent gossip, and others want to claim they grasp nothing because they aren’t full-blooded agents who say only and exactly what they mean.

We have to recognise that most of what humans do, linguistic and otherwise, simply isn’t the more high-bar stuff that many (including me) leverage to distinguish us from mere animals/machines (take your pick). This means we have to be more careful in drawing such distinctions.

This goes double when the high/low bars are blurred. We have built machines that are superhumanly good at gossip. They can gossip at length and in detail about some subjects greater than some humans will ever sincerely explain things they really know.

I’m fond of Kant’s idea that there is no consciousness without the possibility of self-consciousness. What defines self-consciousness here is precisely what I’ve said LLMs lack: something like an integrated world model, and something like intentional agency.

You can read this as denying that there is consciousness/understanding in systems that do not act as full-blooded rational agents, keeping track of their theoretical and practical commitments, giving and asking for reasons, and generally striving for consistency.

You might then treat those systems (animals/machines) as completely different in kind from self-consciousness rational agents, as if they have absolutely nothing in common. But this is the wrong way to look at things.

For the most part inference modulates rather than instigates my behaviour. I say and do what seems appropriate (likely?) in a given context, but I can override these impulses, and even reshape them when they conflict with one another or with my more articulated commitments.

My habitual behaviour is conscious because it can be self-consciously modulated. From one perspective nothing much is added on top of animal cognition, from another, it makes all the difference. This equally applies to my linguistic behaviour.

I often speak without thinking. Maybe I mutter to myself. Maybe I just parrot a wrote phrase. Maybe I say something I don’t really mean. Maybe I don’t even know what it meant. But often it makes sense and is retrospectively judged to have the neat stamp of communicative intent.

Sometimes I gossip. Sometimes I’m just spewing things that sound plausible without much thought, and I’m not paying too much attention to how they fit into my wider rational picture of the world. Sometimes I’m less reliable and informative than an LLM.

I can still be pulled into the game of giving and asking for reasons, forced to full self-consciousness, and thereby put my idle talk (Gerede) to the test. That’s the difference between me and the LLM. The difference that makes the difference. But it doesn’t erase the similarity.

I’m also fond of Hegel’s idea that there is no content without the possibility of expression. I think this is recognisably a development of Kant’s thought. The twist is that it lets us read ‘possibility’ more broadly. We’re no longer confined to an individual consciousness.

We are then capable, for instance, to talk about the content of historically prevalent social norms that have thus far been implicit, making them explicit and critically assessing them in a manner that might accurately be described as a sort of social self-consciousness.

And I suppose this is my point: LLMs are awkwardly entangled in the system of implicit norms that constitute language. They’re incapable of true self-consciousness/expression (rational revision) but they feed into ours in much the way we feed into one another.

They’re in the game, even if they’re not strictly playing it. And that counts for something. It’s enough to maybe say we know what they mean from time to time. Kinda. Perhaps as we do for one another when we’re not quite at our most luminously intentional.

And so, though I do think there are important architectural changes required before we can say *they* know what they mean, these are maybe less extreme than Bender’s arguments would suggest. Best leave it at that.

[image error]

P.S.

As an addendum to yesterday’s big thread on LLMs and meaning, there was an argument I was building towards but didn’t really express regarding the literal/speaker meaning distinction and the dependency between the two.

We can grant this argument if we interpret it to mean that, e.g., it would not make sense to say that LLM outputs were even literally meaningful if they weren’t embedded in a culture of human speakers with communicative intentions.

But I’ve been arguing that we should resist the narrower interpretation that says we can’t say that LLM outputs have literal meaning unless they themselves are capable of fully formed communicative intentions.

I want to add to this the additional claim that there is a dependency running in the other direction. This turns on the idea that possessing communicative intentions requires the capacity for intentional agency more generally.

Now, I have quite strict conception of agency. I think it involves a kind of in principle generality: the ability to form intentions oriented by arbitrary goals. The representation generality of language opens us up to this, allowing us to desire all sorts of ends and means.

There are many people who disagree with me about this, usually because they think animals of various sorts are on an equal agential footing with us, and occasionally because they want to say the same about extant artificial agents.

But even if you disagree with me on the idea that we need some thing like the articulated semantic contents of literal meaning to open us up to the full range of possible intentions, it’s hard to argue that specifically communicative intentions are possible without it.

Speaker intention does not emerge from the animal world fully formed, only waiting for suitable words to express it. There’s obviously a more complex story about how it is bootstrapped, with literal and speaker meaning progressively widening one another’s scope.

But in the artificial realm, it would not be unreasonable for this story to unfold differently, and for the appropriation of an expansive field of literal meanings to form the framework within which it becomes possible to articulate more refined communicative intentions.

And my point in the original thread was that this fairly plausible story is precisely what the post-Searlean obsession with ‘meaning’ forecloses, requiring the perfectly crisp speaker meanings of world-engaged agents to emerge all at once as a fully integrated package.

I see this as the general form of Leibniz’s Mill type arguments, of which Searle’s is exemplary and Bender’s is a distant descendent: a refusal to countenance the possibility of the system’s parts without the whole.

For me, the tricky thing has been to be able to acknowledge that there is something important missing without systemic integration (self-consciousness/rational agency proper), without rejecting potential components witnessed in isolation (semantic scaffolding).

Down that path lies madness, i.e. Searle’s claim that there’s got to be some special property of our parts that allows them to compose the intentionally saturated whole.

[image error]