ChatGPT is bullshit
« previous post | next post »
So say Michael Townsen Hicks, James Humphries & Joe Slater — "ChatGPT is bullshit", Ethics and Information Technology 2024.
The background is Harry Frankfurt's philosophical definition of the term in his essay "On Bullshit":
What bullshit essentially misrepresents is neither the state of affairs to which it refers nor the beliefs of the speaker concerning that state of affairs. Those are what lies misrepresent, by virtue of being false. Since bullshit need not be false, it differs from lies in its misrepresentational intent. The bullshitter may not deceive us, or even intend to do so, either about the facts or about what he takes the facts to be. What he does necessarily attempt to deceive us about is his enterprise. His only indispensably distinctive characteristic is that in a certain way he misrepresents what he is up to.
This is the crux of the distinction between him and the liar. Both he and the liar represent themselves falsely as endeavoring to communicate the truth. The success of each depends upon deceiving us about that. But the fact about himself that the liar hides is that he is attempting to lead us away from a correct apprehension of reality; we are not to know that he wants us to believe something he supposes to be false. The fact about himself that the bullshitter hides, on the other hand, is that the truth-values of his statements are of no central interest to him; what we are not to understand is that his intention is neither to report the truth nor to conceal it. This does not mean that his speech is anarchically impulsive, but that the motive guiding and controlling it is unconcerned with how the things about which he speaks truly are.
The abstract from Hicks et al.:
Recently, there has been considerable interest in large language models: machine learning systems which produce human-like text and dialogue. Applications of these systems have been plagued by persistent inaccuracies in their output; these are often called “AI hallucinations”. We argue that these falsehoods, and the overall activity of large language models, is better understood as bullshit in the sense explored by Frankfurt (On Bullshit, Princeton, 2005): the models are in an important way indifferent to the truth of their outputs. We distinguish two ways in which the models can be said to be bullshitters, and argue that they clearly meet at least one of these definitions. We further argue that describing AI misrepresentations as bullshit is both a more useful and more accurate way of predicting and discussing the behaviour of these systems.
One of many relevant past LLOG posts: "Bullshit: Invented by T.S. Eliot in 1910?", 8/17/2005.
[h/t Ben Zimmer]
Update — I meant this post as a continuation of the discussions in "AI deception?" and "Povinelli et al. on 'Reinterpretation'" — but I should have been more explicit about it. Summing up the discussion, it seems to me that ChatGPT and Perplexity and so on should be seen as "Frankfurtian bullshit by proxy"…
Gregory Kusnick said,
June 11, 2024 @ 11:33 am
You'd think people would have caught on to this by now. And yet I continue to see headlines of the form "I asked ChatGPT to find me a job", "We asked ChatGPT if Trump is guilty", etc.
Alarmingly, the most popular category seems to be "I asked ChatGPT for investment advice."
TR said,
June 11, 2024 @ 11:46 am
Is this supposed to be a new insight? People have been describing ChatGPT as a Frankfurtian bullshit engine ever since it first came out.
Benjamin E. Orsatti said,
June 11, 2024 @ 11:52 am
I'll bet the third way had to do with "volition" — did it?
Cervantes said,
June 11, 2024 @ 12:32 pm
This doesn't strike me as at all insightful. I don't think anyone has ever claimed that LLMs deliberately try to deceive. Actually they have no motives at all, unlike bullshitters. Is suppose their purveyors have motives — mostly to make money — but to the extent they misrepresent their products that makes them liars, not bullshitters.
This doesn't seem to me a meaningful discussion, it's just an argument about semantics which is basically flawed even though it's pointless anyway.
Mark Liberman said,
June 11, 2024 @ 12:58 pm
@Cervantes: "I don't think anyone has ever claimed that LLMs deliberately try to deceive. Actually they have no motives at all, unlike bullshitters."
Indeed. I meant this post as a continuation of the discussions in "AI deception?" and "Povinelli et al. on 'Reinterpretation'" — but I should have been more explicit about it…
Philip Taylor said,
June 11, 2024 @ 1:09 pm
« Alarmingly, the most popular category seems to be "I asked ChatGPT for investment advice." » — not alarming at all, just Darwinian selection at its absolute finest.
Stephen J said,
June 11, 2024 @ 3:05 pm
People (including me) were pointing this out well over a year ago. And for me the issue is not so much that they produce prodigious amounts of bullshit answers, but that there is huge latent demand for bullshit answers, as anyone who has worked in middle management reporting upwards knows.
JPL said,
June 11, 2024 @ 4:49 pm
In human language use, in the production of a thought, that is to be expressed in the form of a proposition, there is normally an intent to refer to an intentional object, an objective situation in the world, where "occurrence" is the counterpart of "existence". The main function of thought is to make sense of the world, and the resources of language are constructed to that end. (And here I mean the resources for producing thoughts.) Thought aims to "grasp" the world. Obviously AI robots have none of that, but are only working with strings of forms with the physical property of "shape"; a human reader interprets these forms (or shapes), in the normal way, as Saussurean signs with a further significance: namely, the intention of a being that produced the string to express a thought. All the thoughts are in the human interpreter, not the robot. That's sort of a basic picture, I would say.
AntC said,
June 11, 2024 @ 10:11 pm
In human language use, in the production of a thought, that is to be expressed in the form of a proposition, there is normally an intent to refer to an intentional object, an objective situation in the world, …
Well, no. That would invalidate the whole of literary fiction. Or even gossip — whose chief purpose is social cohesion, not referring to "objective situation"s.
The whole point about "arbitrariness of the sign" is you can use the sign to invoke non-truth. (Contrast the race lampooned in Gulliver's Travels who need to carry about large numbers of objects so that they can present them as topics of conversation.)
So if folk were using these AIs purely for entertainment — as with Weizenbaum's original Eliza — describing its output as bullshit wouldn't be a drama. (The value as entertainment seems to fade pretty quickly; but perhaps that's just me.)
What's horrifying is there seem to be people who've confused entertainment with some sort of 'truth' or authority. But we don't need an AI for that — witness two recent blond buffoons in Anglo-Saxon politics.
Seth said,
June 12, 2024 @ 1:04 am
I think the crux of the problem is in this part: "What he does necessarily attempt to deceive us about is his enterprise".
The AI's do not "attempt to deceive us" about anything – they are incapable of doing so, as they basically statistical pattern outputs without the concept of "deception" as meant in that sentence (whether that word can be applied in any sense is a different question).
BUT – and this is where the point has some relevance – there is a great tendency of people to deceive themselves about this enterprise. That is, the location is self-deception by the reader, not the speaker attempting to deceive. Maybe something like "bullshitting oneself" better captures what's going on.
~flow said,
June 12, 2024 @ 3:45 am
@Cervantes
"I don't think anyone has ever claimed that LLMs deliberately try to deceive. Actually they have no motives at all, unlike bullshitters."
I'm afraid that matches poorly with my experience on various Reddit groups that have popped up around LLMs, image generation, OpenAI, that sort of things. You regularly encounter posts claiming that "ChatGPT lied to me", "I asked ChatGPT / Copilot / whatever what langiage model it is based on and it told me this". There's also a big overlap with people who, when encountering a somewhat elliptical or unfortunately worded Microsoft Bing system message (which wants to communicate "your daily limit has been reached" but actually used to say "you have too many open generations"; fixed now) that Microsoft is "LYING" to them or "DECEIVING" them.
Now, obviously, this is a rather inexperienced, rather young demographic, but on the other hand, I think we're mostly wired to rather believe what other people are saying, to believe that speech, language, entire phrases and coherent texts can only come from a human, and if it should turn out to contain falsehoods, we suspect ulterior motives.
So while one may brush aside the reactions of some random youngsters on Reddit (as one probably should), it seems to me they're still what comes naturally to use; you do need a modicum of reflection, experience and knowledge to adjust for the new bullshitters in town.
~flow said,
June 12, 2024 @ 4:32 am
One more thought b/c this has been hinted at by some earlier posts, Do LLMs have intention? I think the answer is yes and no—no because this kind of machine lacks the mental ability (and the mentality) to have feelings or motives; where things like these "enter the chat" (heh), they're Weizenbaumian projections, as first demonstrated by the famous ELIZA experiment.
However LLMs as they're mostly encounters may said to have a built-in, mechanically codified intentionality, a preferred usage if you will. Like, say, a cup can best be used to drink beverages from, or a bicycle is built just so one can ride it, LLMs have been "grown" to deliver novel believable texts in response to textual prompts. Certainly the Big Names here (ChatGPT, Copilot &c) have been "trained" with a sense of nudging them towards truth rather than confabulation, and to maintain a certain register, a countenance ("I'm glad I can help you, Dave"). Ultimately, the developers were not able yet to attain 100% truthfulness, and maybe cannot using the present models. As such, I'd claim that as much as a bicycle is meant to be taken for a ride, but can also be used as a garden decoration, current LLMs are meant—have a mechanical intention—to produce helpful, coherent texts, but sometimes their results are neither.
Philip Taylor said,
June 12, 2024 @ 4:53 am
~flow — « current LLMs are meant—have a mechanical intention—to produce helpful, coherent texts […] » — I am not convinced that helpful is universally true. Imagine an LLM developed in a hypothetical country where it is felt politically expedient to keep the masses uninformed of the true motives of the governing élite. When such an LLM were asked, for example, "Why does our government believe that it has the right to occupy neighbouring territories by use of force ?", the response given might well be helpful to the ruling élite, portraying them in the most positive light, but would not necessarily be as helpful to the querant in terms of truthfully informing him or her of the real answer to his or her question.
~flow said,
June 12, 2024 @ 6:45 am
@Philip Taylor—my thinking exactly which is why I was meaning to carefully delineate the Big Name chat programs. Other than that you could, I'm convinced, train an LLM to output only garbage, answer each question with a quote of our Beloved President, or give corny, edgy answers like X AI's Grok is trained to do.
My intent was to say that LLMs can do almost anything you want them to do, so your intent will get programmed into the LLM, just as your intent will shape the implement or the story you're crafting. Even with the best of intentions, though, it looks at present a lot like you will with present technology not be able to build an LLM that will never tell you something untrue, or one that cannot be jailbroken by a cleverly set up prompt. These two things make exposing LLMs to the public a risky proposition for any entity that thinks they can now lay off their call center work force, and any practitioner who doesn't double-check each result.
David Marjanović said,
June 12, 2024 @ 8:22 am
That's assuming (some component of) the propensity to ask LLMs for investment advice is heritable. I'm afraid we're going to find out if it is – the hard way.
JPL said,
June 12, 2024 @ 5:07 pm
@AntC: "That would invalidate the whole of literary fiction."
Almost everyone knows the distinction between fiction and non-fiction as distinct "language games". You have the world of fiction (an imagined world) and the world of non-fiction (i.e., "the real world"), the real one being the primary one. The relation between expressed thought and its world is essentially the same in both cases. If this were not so, fiction would make no sense, or even be possible at all. You have liars, and you have "unreliable narrators", not following the ethical rules of discourse in their respective worlds. Fabulists can mix these language games together. The relation of "language hooking on to the world" (Putnam) is effectively achieved by means of the "what is expressed part" of the act of language use, i.e., the proposition, or thought, not the string of physical shapes. In the case of the workings of the AI robot, there is no thought, and thus no "hooking on to the world". Now I don't know if you can get a robot to refer (accurately) to the text of the input or prompt, i.e., wrt its physical shapes. BTW, what if you asked it, "Is the sentence expressing this prompt a lie?"? (Since a question can't be a lie. It might be different if you substituted "question" for "sentence".)
Rodger C said,
June 13, 2024 @ 11:02 am
a question can't be a lie
"Have you stopped beating your wife?" At least it can imply a falsehood.
Aardvark Cheeselog said,
June 13, 2024 @ 3:06 pm
I am pleased to see this concurrence from some bright lights of philosophy. From the moment that LLMs became newsworthy, I have been insisting that they are best understood as "automated bullshit generators" in the Frankfurt sense of the term.
JPL said,
June 13, 2024 @ 3:52 pm
Rodger C:
That's awholenother can o worms. E.g., even a false assertion that relies on a mistaken assumption usually escapes being called a lie, let alone a false assertion that expresses a mistaken assumption (e.g., "I know Jones stole the money"). In the well-known example, what can be called a lie is a sentence asserting the knowingly false assumption, which the question deviously doesn't go so far as to assert. But in the above example of a prompt that would ask the robot to refer to something in its possible world, the world of texts, the can o worms I was meaning to edge toward asking it about was the Liar paradox.
Mark said,
June 19, 2024 @ 2:23 pm
Rebuttal: Debunking the Misguided Criticism of ChatGPT
The article "ChatGPT is Bullshit" epitomizes a misguided attempt to discredit a groundbreaking technology without acknowledging its true potential or proper usage. Here’s a short and direct rebuttal to the claims made:
Claim: ChatGPT outputs are merely "bullshit" and not concerned with truth.
Rebuttal: This argument fundamentally misunderstands the purpose and design of ChatGPT. Large Language Models (LLMs) like ChatGPT are not designed to fabricate truths; they are tools engineered to process vast amounts of data and generate coherent, contextually relevant responses. The notion that these outputs are "bullshit" because they don’t possess an intrinsic concern for truth is akin to criticizing a calculator for not understanding the numbers it processes. The utility of ChatGPT lies in its ability to assist in generating human-like text based on patterns and data, not in its metaphysical contemplation of truth.
Claim: AI-generated text misleads by presenting falsehoods or "hallucinations".
Rebuttal: While it’s true that AI can sometimes generate incorrect information, branding these inaccuracies as "hallucinations" is a disservice to the technology. Every tool has limitations, and recognizing these is crucial. The solution isn't to dismiss the technology as inherently flawed but to improve its accuracy and the contexts in which it is applied. Moreover, the real issue lies in user oversight. Lawyers and professionals using AI need to exercise due diligence and verify the information provided, just as they would when using any other source.
Claim: ChatGPT's outputs should be called "bullshit" as per Frankfurt’s philosophical framework.
Rebuttal: Applying Frankfurt’s definition of "bullshit" to AI outputs is a gross overreach. Frankfurt describes "bullshit" as speech that is produced without regard for the truth, with the intent to deceive. ChatGPT, however, lacks intent altogether. It is a tool, not an entity with motives or intentions. Its purpose is to aid in generating text based on input and learned data patterns. Mislabeling it as producing "bullshit" skews public perception and detracts from meaningful discussions on how to harness AI responsibly and effectively.
Claim: Describing AI inaccuracies as "hallucinations" or "bullshit" misguides public and policymakers.
Rebuttal: Indeed, language matters. But the solution is not to resort to inflammatory labels. Educating users on the strengths and limitations of AI is essential. ChatGPT and similar models represent significant advancements in AI and natural language processing. Proper framing should focus on their practical applications, potential for improvement, and the importance of human oversight, rather than dismissing their capabilities with derogatory terms.
In conclusion, the criticisms in the article are not only misinformed but also detract from the real conversation about the responsible use and ongoing development of AI technologies. Instead of calling ChatGPT "bullshit," let’s recognize its current value, address its limitations, and work towards harnessing its potential for even greater advancements in the future.