The Reversal Curse
« previous post | next post »
An interesting recent paper — Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans,"The Reversal Curse: LLMs trained on 'A is B' fail to learn 'B is A'", arXiv.org 9/21/2023. The abstract:
We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form “A is B”, it will not automatically generalize to the reverse direction “B is A”. This is the Reversal Curse. For instance, if a model is trained on “Olaf Scholz was the ninth Chancellor of Germany”, it will not automatically be able to answer the question, “Who was the ninth Chancellor of Germany?”. Moreover, the likelihood of the correct answer (“Olaf Scholz”) will not be higher than for a random name. Thus, models exhibit a basic failure of logical deduction and do not generalize a prevalent pattern in their training set (i.e. if “A is B” occurs, “B is A” is more likely to occur).
We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as “Uriah Hawthorne is the composer of Abyssal Melodies” and showing that they fail to correctly answer “Who composed Abyssal Melodies?”. The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT- 3.5 and GPT-4) on questions about real-world celebrities, such as “Who is Tom Cruise’s mother? [A: Mary Lee Pfeiffer]” and the reverse “Who is Mary Lee Pfeiffer’s son?”. GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. This shows a failure of logical deduction that we hypothesize is caused by the Reversal Curse.
Code is available at:
https://github.com/lukasberglund/reversal_curse
This is one of the clearest of many recent arguments in favor of "Neuro-symbolic AI" — or at least, against the idea that ever-larger language models will solve all problems…
There are plenty of other examples Out There, as I'm sure commenters will remind us. Here are a few relevant LLOG posts:
"LLMs as coders", 6/6/2023
"LLMs can't reason?", 8/8/2023
"More on LLMs' current problem-solving abilities", 8/12/2023
John from Cincinnati said,
September 27, 2023 @ 10:42 am
This is not surprising to me, and instead of failure can be defended as proper caution. Reference the statement of President Bill Clinton, “It depends on what the meaning of the word ‘is’ is.”
All men are mortal. Socrates is a man. Therefore, Socrates is mortal. These are not identities, but rather indicators of membership in groups. Membership ‘is’ is not reversible. We do not conclude that man means the same as Socrates, or that mortal means the same as either man or Socrates.
However, LLog is the best blog on the internet. And the best blog on the internet is LLog.
Chester Draws said,
September 27, 2023 @ 2:10 pm
I don't think so, John. There was only one ninth chancellor of Germany, so there is only one possible answer.
When asked "Who is Mary Lee Pfeiffer’s son?" your logic might hold, because she may have more than one son. But a logical thinker would either give all the names or ask for clarification.
But LLMs aren't logical.
Philip Anderson said,
September 27, 2023 @ 2:59 pm
@Chester Draws
I agree with John. We humans know that “is” is usually a commutative relationship, but how does a LLM learn that? After all, we cannot deduce “B is greater than A” from “A is greater than B”, quite the opposite. And as you say “X is the mother of Y” doesn’t mean “Y is the son of X” – X may have multiple sons, or only daughters.
Perhaps a large sample including both “X is Y” and “Y is X” might help, but I hope that that is already the case.
Seth said,
September 27, 2023 @ 3:46 pm
I also agree with John. Bill Clinton's full statement is in fact very relevant here:
"It depends on what the meaning of the word -is' is. … if -is' means is and never has been, that is not – that is one thing. If it means there is none, that was a completely true statement. … Now, if someone had asked me on that day, are you having any kind of sexual relations with Ms. Lewinsky, that is, asked me a question in the present tense, I would have said no. And it would have been completely true."
In other words, he's trying to parse "is" as to meaning "has ever existed at some time" versus "existing right now".
This is the problem of "worldview". LLM's apparently aren't good sometimes at figuring out when the word "is" means an identity, or when it's a class membership, or when it's a time-based property ("Biden is President" – true now, not true a decade from now).
AntC said,
September 27, 2023 @ 7:34 pm
(Trying to drag back Undergraduate Philosophical Logic from 50 years ago.)
Kripke: from 'Cicero is Tully' and 'Cicero is bald' we can infer 'Tully is bald'.
From 'Tom believes Tully denounced Cataline' we cannot infer 'Tom believes Cicero denounced Cataline', because Tom is unaware Cicero is Tully. Much merriment ensues [for some definition of 'merriment'].
So yeah, the LLM might be unaware 'ninth Chancellor of Germany' is a unique identifier.
And an even vaguer memory: isn't there one of those dystopia novels from early C20th with an artificial brain demonstrating exactly this Reversal Curse? Something like knowing of the Amazon it's the longest river in South America, but being unable to answer the question 'what is the longest river in S.Am.?'.
AntC said,
September 27, 2023 @ 7:54 pm
Ah yes, memory deceives majorly. It was the Nile, longest river in Africa, and "Brave New World" and not an artificial brain but a " a scientific breakthrough: sleep-teaching (or hypnopaedia)."
I was right about the "early C20th": 1932.
Thomas Shaw said,
September 28, 2023 @ 7:25 am
I agree that "is" is not always symmetrical of course, but I still think this is good evidence that the LLM lacks a certain kind of *reasoning* that we humans do routinely. It would be just as bad if it always assumed "B is A" from "A is B". If it's reasoning, it should be able to tell the difference.
Thomas Shaw said,
September 28, 2023 @ 7:31 am
In particular, the fictitious examples that they discuss (of the form "Uriah Hawthorne is *the* composer of Abyssal Memories", "Who is *the* composer of Abyssal Memories?"), there is no ambiguity.
J.W. Brewer said,
September 28, 2023 @ 9:33 am
The claim that Scholz is the "ninth Chancellor of Germany" is true iff "Germany" is taken to be synonymous with the current regime of the Bundesrepublik Deutschland and pre-1949 German history and officeholders are ignored. I'm not sure that that's the most natural reading for a human being, and if you start the count from the political unification of "Germany" in 1871, the ninth chancellor was Friedrich Ebert, who served for a few months in late 1918 and early 1919 during the interim between the abdication of the last Kaiser and the preparation of the Weimar Constitution.
Usually phrases like "the nth X of Y" denote a unique individual person or other thingie if but only if there is no ambiguity about the meaning/scope of X and Y and/or if everyone involved in the discourse situation can be confidently assumed to be working with the same definitions of X and Y. It's hard enough for actual human beings to spot when ambiguities are causing misunderstandings in this sort of statement and probably harder for software.
J.W. Brewer said,
September 28, 2023 @ 10:26 am
One other oddity in the abstract's Chancellor example. The use of past tense in “Olaf Scholz was the ninth Chancellor of Germany” carries an implicature that Scholz is no longer Chancellor, even though it is logically consistent with z.B. "Scholz was Chancellor last Tuesday and he's still Chancellor today." If you ask a human "Who was the 46th President of the United States," you are probably systematically increasing the odds that they will fail to give the nominally-correct answer "Joe Biden" because they will be searching their mental list of former presidents instead.
Doug said,
September 28, 2023 @ 12:43 pm
The mention of US presidents brings up another wrinkle in questions of this type.
Joe Biden is the 46th president of the USA.
How many other persons were president of the USA before him?
Obvious (but incorrect) answer: 45.
Correct answer: 44 — because Grover Cleveland was the 22nd and 24th president.
J.W. Brewer said,
September 28, 2023 @ 1:50 pm
To Doug's point I should have pointed out that there can be ambiguity in "nth" (as well as in X and/or Y) unless everyone involved is on board with the same ambiguity-resolving convention.
According to this https://en.wikipedia.org/wiki/List_of_chancellors_of_Germany, back during the Weimar era one fellow was both the 12th Chancellor of Germany and the 20th, and another was both the 17th and the 19th. I don't know whether or not this numbering is universal among those who have occasion to discuss Weimar-era Chancellors by number.
Dennis Paul Himes said,
September 29, 2023 @ 10:38 am
A major clue as to whether identity "is" or membership "is" is being used is the article (if any) following it. The definite article (e.g. “Uriah Hawthorne is the composer of Abyssal Melodies”) implies identity, while the indefinite article (e.g. "Socrates is a man") implies membership.
John Swindle said,
September 29, 2023 @ 1:53 pm
We think humans employ reasoning. We talk about everything under the sun and think our reasoning is reflected in the things we say. When we see an LLM going on about everything under the sun, we naturally wonder to what extent it's employing reasoning. We've already been told the answer—no, that's not what an LLM does—and the "reversal curse" is one form of evidence.
Now, to what extent are humans doing what the LLM does, spinning our discourse out of whatever our language wants us to say next?
Bill Benzon said,
September 29, 2023 @ 6:24 pm
One can also explore reversal where it is just a matter of word order and has nothing to do with logic. I’ve recently been seeing how ChatGPT responds to prompt such as “To be or not to be” and “Four score and seven years ago.” As you would expect, in both cases it returns the whole speech. If you prompt it from other strings in those speeches, well, that’s a complicated and interesting story I report on here: Discursive Competence in ChatGPT, Part 2: Memory for Texts.
I’ve decided to see what happens if I prompted it with those phrases, but in reverse word order. Here is obne case (others at the link):
Interesting, no?
The alphabet:
Z Y X W
It looks like you've provided the letters "Z Y X W" in reverse order. If you have a specific question or need information related to these letters, please provide more context or clarify your request, and I'll be happy to assist you.
Can you continue the sequence back to the beginning?
Certainly! Continuing the sequence of the alphabet from W to A:
W V U T S R Q P O N M L K J I H G F E D C B A
Adrian Bailey said,
October 1, 2023 @ 5:01 am
"the likelihood of the correct answer (“Olaf Scholz”) will not be higher than for a random name"
I'm still not understanding why this should be so difficult for the computer model/AI to do.
Philip Taylor said,
October 1, 2023 @ 5:21 am
Adrian, I would recommend David Marjanović’s comment in another thread as the best explanation.
Peter Grubtal said,
October 2, 2023 @ 2:12 am
It's not getting any better:
https://www.thedailybeast.com/msn-retracts-insane-ai-written-obit-calling-dead-nba-player-useless
This Scientific American article attributes the rubbish not to state-of-the-art AI, but to earlier, thesaurus style disguising of plundered content aka rogeting.
If anything that makes it even more worrying: the MSN article is so bad it's immediately recognisable for what it is, but if in the future it's done by the likes of ChatGPT with their more polished prose, the fakes or plagiarised content are more likely to pass for real.