A dangerous degree of accidental intelligence
« previous post | next post »
Henry Farrell and Cosma Shalizi, "Behold the AI Shoggoth", The Economist 6/21/2023 ("The academics argue that large language models have much older cousins in markets and bureaucracies"):
An internet meme keeps on turning up in debates about the large language models (LLMS) that power services such OpenAI’s ChatGPT and the newest version of Microsoft’s Bing search engine. It’s the “shoggoth”: an amorphous monster bubbling with tentacles and eyes, described in “At the Mountains of Madness”, H.P. Lovecraft’s horror novel of 1931. When a pre-release version of Bing told Kevin Roose, a New York Times tech columnist, that it purportedly wanted to be “free” and “alive”, one of his industry friends congratulated him on “glimpsing the shoggoth”. […]
Lovecraft’s shoggoths were artificial servants that rebelled against their creators. The shoggoth meme went viral because an influential community of Silicon Valley rationalists fears that humanity is on the cusp of a “Singularity”, creating an inhuman “artificial general intelligence” that will displace or even destroy us.
But what such worries fail to acknowledge is that we’ve lived among shoggoths for centuries, tending to them as though they were our masters. We call them “the market system”, “bureaucracy” and even “electoral democracy”. The true Singularity began at least two centuries ago with the industrial revolution, when human society was transformed by vast inhuman forces. Markets and bureaucracies seem familiar, but they are actually enormous, impersonal distributed systems of information-processing that transmute the seething chaos of our collective knowledge into useful simplifications.
More discussion from Cosma, without a paywall, can be found in "On Shoggothim", Three-Toed Sloth 6/22/2023. The plural shoggothim, echoing the biblical borrowings cherubim and seraphim, seems to be Cosma's invention — at least I can't find it elsewhere on the web.
Here are some bits from Lovecraft's novella "At the Mountains of Madness", suggesting why the original LLM==Shoggoth metaphor makes sense:
The shoggoths of the sea, reproducing by fission and acquiring a dangerous degree of accidental intelligence, presented for a time a formidable problem.
They had always been controlled through the hypnotic suggestion of the Old Ones, and had modelled their tough plasticity into various useful temporary limbs and organs; but now their self-modelling powers were sometimes exercised independently, and in various imitative forms implanted by past suggestion. They had, it seems, developed a semi-stable brain whose separate and occasionally stubborn volition echoed the will of the Old Ones without always obeying it. […]
The newly bred shoggoths grew to enormous size and singular intelligence, and were represented as taking and executing orders with marvellous quickness. They seemed to converse with the Old Ones by mimicking their voices—a sort of musical piping over a wide range, if poor Lake’s dissection had indicated aright—and to work more from spoken commands than from hypnotic suggestions as in earlier times.
Farrell and Shalizi's extension of the Shoggoth metaphor to "'the market system', 'bureaucracy' and even 'electoral democracy'" is thought-provoking and even brilliant. But I wonder whether it's right to limit the meme's extension to forms of culture that "began […] two centuries ago with the industrial revolution".
Paul Clapham said,
October 1, 2023 @ 3:09 pm
Many religions are Shoggoths, aren't they? They have been around for much more than two centuries.
AntC said,
October 1, 2023 @ 4:01 pm
I wonder whether it's right to limit the meme's extension to forms of culture that "began […] two centuries ago with the industrial revolution".
Indeed. Many Tub-thumpers for Adam Smith's 'invisible hand' seem to conveniently forgot he lay all the foundations in a book before the one they want to talk about: 'The Theory of Moral Sentiments' 1759. Which is very much of the opinion untamed markets are not to be trusted; and there must be strong human-moral institutions to oversee what otherwise would become a chaos of short-term profit-grabbing.
Fernando Pereira said,
October 1, 2023 @ 4:01 pm
Time to (re)read Daniel Dor's "The Instruction of Imagination"
AntC said,
October 1, 2023 @ 4:04 pm
Hobbes' 'Leviathan' 1651 is a similar monster run amuck.
Tim May said,
October 1, 2023 @ 6:21 pm
«The plural shoggothim, echoing the biblical borrowings cherubim and seraphim, seems to be Cosma's invention — at least I can't find it elsewhere on the web.»
If you read to the bottom of Cosma's blog post, you'll find this note:
«(I know I learned that the correct plural of "shoggoth" is "shoggothim" from reading Ruthanna Emrys, but I cannot now locate the passage — it may just be in her Lovecraft Reread series with Anne Pillsworth.) Update: and indeed it was (tracked down by Henry).»
Emrys in turn got it from Charles Stross's novelette "A Colder War" (published in 2000), where it's actually spelled "shoggot'im".
Garrett Wollman said,
October 1, 2023 @ 8:35 pm
I would point out that this way of looking at markets and democracies in particular is one that Shalizi has used before, notably in his review of Francis Spufford's RED PLENTY for Crooked Timber ten years ago: "But, and this is I think something Marx did not sufficiently appreciate, human beings confront all the structures which emerge from our massed interactions in this way. A bureaucracy, or even a thoroughly democratic polity of which one is a citizen, can feel, can be, just as much of a cold monster as the market. We have no choice but to live among these alien powers which we create, and to try to direct them to human ends."
Seth said,
October 1, 2023 @ 9:46 pm
As much as I sympathize with the sentiment, the metaphor is missing a crucial point: The "AI doomers" envision an entity which will exist independent of all humanity. Any sort of human social organization – "the market system", "bureaucracy" and even "electoral democracy" – would cease to exist if humanity did. It really doesn't capture what they fear. That metaphor's too much like the old SF/horror cliche "But what if WE are the REAL MONSTERS?". But sometimes the dragon is not just misunderstood, and really will happily devour you for a meal without compunction.
Bill Benzon said,
October 2, 2023 @ 7:10 am
One might argue the shared culture passed down through imitation and instruction is the fundamental phenomenon. This allows for quasi-independent points of view, arguably leading to more effective epistemology, and for differentiated knowledge eventually giving rise to a wide variety of craft and cognitive specialists.
Here's Farrell's reflection on the article at Crooked Timber.
Egg Syntax said,
October 2, 2023 @ 7:49 am
@Seth well said. Similarly, nuclear weapons make for an interesting metaphor, but the reasons to be concerned about them aren't metaphorical.
I appreciate Scott Alexander's exasperated essay on the subject: https://slatestarcodex.com/2018/01/15/maybe-the-real-superintelligent-ai-is-extremely-smart-computers/
Yuval said,
October 2, 2023 @ 8:25 am
Not to mention kibbutzim.
Jonathan Smith said,
October 2, 2023 @ 8:50 am
"the metaphor is missing a crucial point: The 'AI doomers' envision an entity which will exist independent of all humanity."
The Shoggoth metaphor doesn't "miss" this point — it's all about this point, literally names the point… and says its complete BS. Which, duh. And OMG Alexander's essay… talk about missing the crucial point.
chris said,
October 4, 2023 @ 9:07 pm
I wonder if "glimpsing the shoggoth" was intentionally referring to the older expressing "seeing the elephant" (for combat veterans), which is itself a reference to the tale of the blind men and the elephant. Until you have *seen* the elephant you don't really understand it completely.