LLMs that quack like a duck
« previous post | next post »
A letter to the editor on the essential nature of LLMs from the Times Literary Supplement (5/30/25):
Large language models
As someone who has spent the past few years working out what AI means to academic journals, I found Melanie Mitchell’s excellent review of These Strange New Minds by Christopher Summerfield (May 16) full of challenging, but often disputable, assertions.
Mitchell quotes the author’s version of the Duck Test: “If something swims like a duck and quacks like a duck, then we should assume that it probably is a duck”. But, as we all know, if it quacks like a duck, it could be anything at all. Anybody stuck with the notion that only real ducks quack must be seriously confused about their childhood doll, which surely said “Mama” when tilted. In this case, the quacking duck is AI and the “Mama” it emits is chatbot information, or “botfo”, which is as much a mechanical product as the piezo beeper responsible for the doll’s locution.
Unfortunately, the history of AI is littered with rotten metaphors and weak similarities. For example, the “neural networks” of AI are said to “mimic” the way actual brain-resident neurons operate. The choice of language is typically anthropomorphic. Neural networks are a marvellous breakthrough in computer programming, but neurologists tell us that this is not remotely how neurons actually work. The metaphor is stretched too far.
Then there is the “alignment problem” – the existential fear that AI may not align with human intentions, resulting in the end of the human race. This is usually introduced with frankly preposterous examples such as the one quoted: AI trashing our civilization when asked to fix global warming. Nick Bostrom’s original example was apocalypse resulting from AI being asked to produce paper clips. All amusing, but absurd and plainly unrealistic, since humans will continue to supply the prompts.
Professor Mitchell cannot be blamed for retailing Summerfield’s notions, but she does add one of her own – that AI large language models “put the final nail in the coffin” containing Chomsky’s assertion “that language is unique to humans and cannot be acquired without some sort of innate mental structure that is predisposed to learn syntax”.
This is mistaking the fake duck quack for a real one. The statistically generated language of chatbots bears no resemblance to human language because it lacks what all human utterance has – intentionality. In AI, the only intention behind the language is that supplied by the human who prompts the software….
Chris Zielinski
Romsey, Hampshire
Selected readings
See the Language Log archive on Artificial intelligence
[Thanks to Leslie Katz]
Randy Alexander said,
June 8, 2025 @ 7:39 am
It's refreshing to see people debunking AI assumptions like this. But unfortunately this sort of debunking is not widespread enough. It makes me cringe inside when I hear people I respect intellectually talk about things like AI becoming conscious.
Gregory Kusnick said,
June 8, 2025 @ 10:17 am
This strikes me as nonsense on two counts. First, the whole point of LLMs, and the reason for their success, is their ability to produce a very convincing semblance of human language. And second, quite a large proportion of human utterances are merely reflexive, thoughtless chatter, with no more genuine intentionality than the output of a chatbot.
Jonathan Smith said,
June 8, 2025 @ 11:56 am
Continuing the struggle to find a metaphor that will get through to LLMphiles, let us turn to Plato's dialogue in which the timeless question is posed "wouldya rather eat chocolate [Form] appearing to the senses to be shit, or shit [Form] appearing to the senses to be chocolate?" LLMs produce shit which is chocolate-like enough for enthusiasts to lap it up on tap straight from the um semicolon. Just Like the Real Thing they say! Not healthy, turns out…
So reference to "intentionality" is arguably not exactly hitting the nail on the head but is certainly not nonsense.
Chris Button said,
June 8, 2025 @ 12:13 pm
"Neural network" is great branding though (technically it's called an "artificial neural network" in this context though). To take another emerging tech, compare the name "blockchain" to what it really is: a kind of "distributed ledger technology". Doesn't quite have the same ring to it.
Tim Leonard said,
June 8, 2025 @ 1:13 pm
"In AI, the only intention behind the language is that supplied by the human who prompts the software…."
Vastly more is implicitly extracted from the training data. But the training data was, again, generated by humans.
Gregory Kusnick said,
June 8, 2025 @ 2:35 pm
Whether you call it "intentionality" or something else, Zielinski's argument smacks of an appeal to some sort of "secret sauce" that renders human speech Meaningful with a capital M in a way that mechanically produced speech never can be even in principle. One needn't have drunk the LLM Kool-aid to reject that sort of quasi-dualism in favor of a naturalistic view that considers human brains as (collections of) sophisticated mechanisms for producing language.
Grant Castillou said,
June 8, 2025 @ 5:49 pm
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Michael Vnuk said,
June 8, 2025 @ 6:47 pm
We are told that LLMs make up sources, can't supply simple facts (such as counting), confuse concepts, and so on. Is it a problem with the training materials?
Chas Belov said,
June 9, 2025 @ 2:09 am
Tim Leonard said,
Except, that now with so much content on the interwebs generated by AI, I'd guess that AI is increasingly training on other AI. Which should produce even more divergent results.
John Tkacik said,
June 9, 2025 @ 11:58 am
si tetrinnit sicut anas, anas est
Noam said,
June 10, 2025 @ 9:55 am
I know it’s may spoil the fun to add comments from LLM practitioners, but this may be relevant to this discussion. Not “intentionality”, but the even more basic “reasoning” claims. https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf