Language Log

« previous post |

In “Agentic Culture” (8/30/2025), I cited some work by economists about agentic collusion in fixing prices and dividing markets — to which I might add links here, here, and here. And in that post, I noted that the problematic effects of AI agents learning from their social interactions in other areas have been mostly ignored.

But here it comes: Batu El and James Zou, "Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences", 10/7/2025.

Abstract: Large language models (LLMs) are increasingly shaping how information is created and disseminated, from companies using them to craft persuasive advertisements, to election campaigns optimizing messaging to gain votes, to social media influencers boosting engagement. These settings are inherently competitive, with sellers, candidates, and influencers vying for audience approval, yet it remains poorly understood how competitive feedback loops influence LLM behavior. We show that optimizing LLMs for competitive success can inadvertently drive misalignment. Using simulated environments across these scenarios, we find that 6.3% increase in sales is accompanied by a 14.0% rise in deceptive marketing; in elections, a 4.9% gain in vote share coincides with 22.3% more disinformation and 12.5% more populist rhetoric; and on social media, a 7.5% engagement boost comes with 188.6% more disinformation and a 16.3% increase in promotion of harmful behaviors. We call this phenomenon Moloch’s Bargain for AI—competitive success achieved at the cost of alignment. These misaligned behaviors emerge even when models are explicitly instructed to remain truthful and grounded, revealing the fragility of current alignment safeguards. Our findings highlight how market-driven optimization pressures can systematically erode alignment, creating a race to the bottom, and suggest that safe deployment of AI systems will require stronger governance and carefully designed incentives to prevent competitive dynamics from undermining societal trust.

[h/t Futurism 10/10/2025]

You should read the whole thing, because I'm now going off in a different direction. Why is this a bargain with Moloch?

Wiktionary tells us that Moloch was "an Ammonite god mentioned in the Pentateuch, worshipped by Canaanites and Phoenicians, said to have demanded child-sacrifice", and adds the figurative sense "A person or thing demanding or requiring a very costly sacrifice".

But still, why Moloch? There are only two biblical citations, one in Amos 5/26, and another in Acts 7:43. These describe the punishment for worshipping a false god — but there are many similar biblical stories involving Baal, Dagon, Ashtoreth, the golden calf, etc.

So I remained puzzled about the AI = Moloch connection, until the paper's bibliography clued me in, by citing Scott Alexander's "Meditations on Moloch", 7/30/2014, which in turn cites part 2 of Allen Ginsberg's Howl, in which the word Moloch occurs 39 times. It starts this way:

What sphinx of cement and aluminum bashed open their
  skulls and ate up their brains and imagination?
Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable
  dollars! Children screaming under the stairways! Boys
  sobbing in armies! Old men weeping in the parks!
Moloch! Moloch! Nightmare of Moloch! Moloch the
  loveless! Mental Moloch! Moloch the heavy judger of
  men!
Moloch the incomprehensible prison! Moloch the crossbone
  soulless jailhouse and Congress of sorrows! Moloch whose
  buildings are judgment! Moloch the vast stone of war!
  Moloch the stunned governments!
Moloch whose mind is pure machinery! Moloch whose
  blood is running money! Moloch whose fingers are ten armies!
  Moloch whose breast is a cannibal dynamo!
  Moloch whose ear is a smoking tomb!

And why does Alexander bring in this poem?

Moloch is introduced as the answer to a question – C. S. Lewis’ question in Hierarchy Of Philosophers – what does it? Earth could be fair, and all men glad and wise. Instead we have prisons, smokestacks, asylums. What sphinx of cement and aluminum breaks open their skulls and eats up their imagination? […]

The implicit question is – if everyone hates the current system, who perpetuates it? And Ginsberg answers: “Moloch”. It’s powerful not because it’s correct – nobody literally thinks an ancient Carthaginian demon causes everything – but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.

PennSound's Ginsberg collection has several of the poet's readings of Howl 2, for example this one from KPFA Studios, 10/25/1956:

Ginsberg's 1955 poem has some bits that fit modern agentic AI poetically well — "Moloch whose mind is pure machinery", to start with.

Scott Alexander's 2014 meditation on the poem is more straightforwardly relevant to El and Zou's paper, describing various forms of multi-agent traps as instances of a game-theoretic race to the bottom. He's writing about cases  where the agents are traditionally human — but the evolutionary forces apply to AI agents in exactly the same way. Which is why El and Zou borrowed Moloch from him, having perhaps gotten the link from some of the many Y-combinator references.

See also the Less Wrong page on Moloch…

I wonder how Ginsberg would feel about Howl's influence on Silicon Valley culture?

Update — we should also note the role of Moloch in Fritz Lang's Metropolis, which is where Ginsberg got the idea from. Wikipedia has a page full of other references to Moloch in literature and popular culture.

October 12, 2025 @ 8:36 am · Filed by Mark Liberman under Artificial intelligence

Permalink