"Moloch's bargain"?
« previous post | next post »
In “Agentic Culture” (8/30/2025), I cited some work by economists about agentic collusion in fixing prices and dividing markets — to which I might add links here, here, and here. And in that post, I noted that the problematic effects of AI agents learning from their social interactions in other areas have been mostly ignored.
But here it comes: Batu El and James Zou, "Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences", 10/7/2025.
Abstract: Large language models (LLMs) are increasingly shaping how information is created and disseminated, from companies using them to craft persuasive advertisements, to election campaigns optimizing messaging to gain votes, to social media influencers boosting engagement. These settings are inherently competitive, with sellers, candidates, and influencers vying for audience approval, yet it remains poorly understood how competitive feedback loops influence LLM behavior. We show that optimizing LLMs for competitive success can inadvertently drive misalignment. Using simulated environments across these scenarios, we find that 6.3% increase in sales is accompanied by a 14.0% rise in deceptive marketing; in elections, a 4.9% gain in vote share coincides with 22.3% more disinformation and 12.5% more populist rhetoric; and on social media, a 7.5% engagement boost comes with 188.6% more disinformation and a 16.3% increase in promotion of harmful behaviors. We call this phenomenon Moloch’s Bargain for AI—competitive success achieved at the cost of alignment. These misaligned behaviors emerge even when models are explicitly instructed to remain truthful and grounded, revealing the fragility of current alignment safeguards. Our findings highlight how market-driven optimization pressures can systematically erode alignment, creating a race to the bottom, and suggest that safe deployment of AI systems will require stronger governance and carefully designed incentives to prevent competitive dynamics from undermining societal trust.
[h/t Futurism 10/10/2025]
You should read the whole thing, because I'm now going off in a different direction. Why is this a bargain with Moloch?
Wiktionary tells us that Moloch was "an Ammonite god mentioned in the Pentateuch, worshipped by Canaanites and Phoenicians, said to have demanded child-sacrifice", and adds the figurative sense "A person or thing demanding or requiring a very costly sacrifice".
But still, why Moloch? There's a bunch of biblical citations — five in Leviticus, four in Kings, and one each in Jeremiah, Amos, and Acts. These describe the consequences of worshipping a false god — but there are similar biblical stories involving Baal, Dagon, Ashtoreth, the golden calf, etc.
So I remained puzzled about the AI = Moloch connection, until the paper's bibliography clued me in, by citing Scott Alexander's "Meditations on Moloch", 7/30/2014, which in turn cites part 2 of Allen Ginsberg's Howl, in which the word Moloch occurs 39 times. It starts this way:
What sphinx of cement and aluminum bashed open their skulls and ate up their brains and imagination? Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks! Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men! Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments! Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!
And why does Alexander bring in this poem?
Moloch is introduced as the answer to a question – C. S. Lewis’ question in Hierarchy Of Philosophers – what does it? Earth could be fair, and all men glad and wise. Instead we have prisons, smokestacks, asylums. What sphinx of cement and aluminum breaks open their skulls and eats up their imagination? […]
The implicit question is – if everyone hates the current system, who perpetuates it? And Ginsberg answers: “Moloch”. It’s powerful not because it’s correct – nobody literally thinks an ancient Carthaginian demon causes everything – but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.
PennSound's Ginsberg collection has several of the poet's readings of Howl 2, for example this one from KPFA Studios, 10/25/1956:
Ginsberg's 1955 poem has some bits that fit modern agentic AI poetically well — "Moloch whose mind is pure machinery", to start with.
Scott Alexander's 2014 meditation on the poem is more straightforwardly relevant to El and Zou's paper, describing various forms of multi-agent traps as instances of a game-theoretic race to the bottom. He's writing about cases where the agents are traditionally human — but the evolutionary forces apply to AI agents in exactly the same way. Which is why El and Zou borrowed Moloch from him, having perhaps gotten the link from some of the many Y-combinator references.
See also the Less Wrong page on Moloch…
I wonder how Ginsberg would feel about Howl's influence on Silicon Valley culture?
Update — we should also note the role of Moloch in Fritz Lang's Metropolis, which is where Ginsberg got the idea from. Wikipedia has a page full of other references to Moloch in literature and popular culture.
Rodger C said,
October 12, 2025 @ 9:45 am
I don't recall C. S . Lewis writing anything called Hierarchy of Philosophers. Perhaps C. I. Lewis is meant?
And neither of them (rather Clifford Bax) wrote the fine hymn "Turn Back, O Man, Forswear Thy Foolish Ways," which is alluded to here.
Mark Liberman said,
October 12, 2025 @ 11:05 am
@Roger C "I don't recall C. S . Lewis writing anything called Hierarchy of Philosophers. Perhaps C. I. Lewis is meant?":
I was unable to find any work by that name, regardless of author. Alexander provides a URL, which currently leads nowhere; but the Wayback Machine gives us a stack of philosophers' answers to progressively abbreviated questions:
So I think it was a joke.
jhh said,
October 12, 2025 @ 11:24 am
The Wikipedia article "Moloch" is quite good at tracing references to Moloch through popular culture. Not to be forgotten is William Blake…
kworks said,
October 12, 2025 @ 1:36 pm
Thought to share a link to a lecture by the novelist Russell Banks called [Feeding Moloch: The Sacrifice of Children on the Altar of Capitalism]
(https://www.youtube.com/watch?v=Kv3Rbq5WIK4&t=1036s) from 10 years ago.
JPL said,
October 12, 2025 @ 3:54 pm
Here's a paper I would like to see: "The role of ideal principles in the emergence of normative systems". I remember a young man at the beginning of his career telling me, "The most important thing in life is how much money you have.", and then going on, in response to my disagreement, to elaborate on what he felt called upon to do as an actor in a capitalist system.
Julian said,
October 12, 2025 @ 7:20 pm
Editor here.
"…on social media, a 7.5% engagement boost comes with 188.6% more disinformation…'
Doncha love the spurious precision?
The presence of a sentence like that in an academic article makes it 73.8% more likely that this reader will want to get out and do some gardening.
Seriously though, not wanting to sound snide, I'd welcome any comment from more expert folks: Is this article a sound and worthwhile addition to the relevant literature?
If so, how much trouble would it be to express its key points in language that's comprehensible outside the priesthood?
Genuinely curious. I can't begin to form any opinion on these questions myself as I'm not in the priesthood.
By the way, I read the referenced Coase 1960 article The Problem of Social Cost. It's admirably clear (except that he does go on a bit and seems to be allergic to the new paragraph marker), and it doesn't include a single Greek letter or pseudomathematical expression.
Joe said,
October 12, 2025 @ 7:35 pm
Mark: Scott Alexander Siskind (he writes under his real-life name now) is very popular in the AI alignment community, inasmuch as it overlaps TESCREAL, so that's almost certainly where they heard of Moloch and which version of Moloch they heard of.
Julian: I recoil a little bit when people put a decimal place in a percentage, because it's just ironic. The whole point of a percentage is that we arbitrarily decided 100 is a good number of units for subdividing something, the right amount of precision rather than 10 or 1000. If you need a scale with 1000 units on it, then theoretically you should just use per mil instead of per cent, so we'd write "75‰" instead of "7.5%". Theoretically "7.5%" should never be written. But maybe before you ask yourself whether your analysis really needs that extra decimal place of precision, you should ask yourself whether your estimate is really that precise in the first place.
Peter Cyrus said,
October 13, 2025 @ 5:22 am
The multiple references to "alignment" caught my attention. If I'd encountered it before in this sense, I hadn't noticed it (although https://en.wikipedia.org/wiki/Alignment_(Dungeons_%26_Dragons)).
I guess it means "good-ness", but I like it because it implies that others may sensibly judge what you find good to be bad.
Kenny Easwaran said,
October 13, 2025 @ 1:11 pm
Although Scott Alexander is no longer entirely pseudonymous, he still prefers to go by "Scott Alexander" when one is talking about his blog – he understandably wants to try to make sure that his two professional lives stay separate. He no more writes under the name "Scott Alexander Siskind" than the musicians record under the names "Lady Gaga Germanotta" or "Madonna Ciccone".
Brett said,
October 15, 2025 @ 10:10 pm
@Joe: I knew one scientist (a very eminent British-American climatologist) who consistently used ‰. It wasn't too bad in written form, but the confusion that arose when he read it aloud as "per mil" was huge.