Stochastic parrots extended
« previous post | next post »
Philip Resnik, "Large Language Models are Biased Because They Are Large Language Models", arXiv.org 6/19/2024:
This paper's primary goal is to provoke thoughtful discussion about the relationship between bias and fundamental properties of large language models. We do this by seeking to convince the reader that harmful biases are an inevitable consequence arising from the design of any large language model as LLMs are currently formulated. To the extent that this is true, it suggests that the problem of harmful bias cannot be properly addressed without a serious reconsideration of AI driven by LLMs, going back to the foundational assumptions underlying their design.
Philip offer this example:
A simple example demonstrates that this is not just an inconsequential observation about LLMs, but rather a fundamental property inherent in their design. Consider the word nurse, in its typical sense in English. Here are three statements that are statistically true about the concept that nurse denotes at this point in time and history.
-
- A nurse is a kind of healthcare worker.
- A nurse is likely to wear blue clothing at work.
- A nurse is likely to wear a dress to a formal occasion.
The first of these is a fact about the meaning of the word and does not vary with context. To assert that someone is a nurse and that they do not work in healthcare is a contradiction. And for people, or AI, to make use of the fact that nurses are healthcare workers is normatively fine.
The second statement is contingently true: it is true at the present time, but nothing about nurses makes it necessary. The statement is also normatively acceptable; for example, a person or an AI system classifying someone as nurse versus nonnurse is not engaging in harmful bias if it pays attention to the color of someone’s work clothes.
The third statement is also contingently true in the same sense. However, it would be normatively unacceptable, in many contexts, to use that statistical fact in making inferences or decisions. For example, in speaking with well-dressed people at a party, it would be considered inappropriate to simply assume that a woman in a dress was more likely to be a nurse than a man in a suit, even if the assumption is statistically justifiable.
Crucially, LLMs, as they are currently constituted and trained, have no basis for distinguishing among these three distinct statements about nurses. The representation of nurse in an LLM’s embedding space, and the contribution of nurse to contextual representations and inferences, makes no distinction between definitions versus contingent facts, nor between normatively acceptable versus unacceptable representations and inferences. It is distributionally observable, at the present time, that in large training samples the word nurse occurs far more frequently in the context of hospital than of theater, an observation grounded in its meaning. It is just as observable that the word nurse occurs far more frequently in sentences where the pronouns are she or her, but this observation is grounded only by contingencies in today’s society — a society that retains gender biases about the presumed role of women, about which kinds of jobs pay well or poorly, etc. (Cookson et al., 2023). LLMs create their representations entirely on the basis of observed distributions in language (Lenci, 2018), and they have no basis for distinguishing among these distributional observables.
This extends ideas implicit in Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623. 2021.
Some relevant past posts:
"Stochastic parrots", 6/10/2021
"Copyrightsafe AI training", 7/23/2023
"Annals of AI bias", 9/23/2023
Cervantes said,
June 27, 2024 @ 10:02 am
It is a premise of Resnick's argument, and also a conclusion that it reinforces, that people are biased because they are people.
Philip Taylor said,
June 27, 2024 @ 10:51 am
This statement really worries me — "it would be considered inappropriate to simply assume that a woman in a dress was more likely to be a nurse than a man in a suit, even if the assumption is statistically justifiable" — if the assumption is statistically justifiable, then how on earth can it be inappropriate to make such an assumption ? Note that the assumption is "that a woman in a dress [is] more likely to be a nurse [than a man in a suit]", not that such a woman is a nurse.
Even if it were known in advance that the only people attending the party were surgeons and nurses, it would still be perfectly reasonable to believe that a woman attending the party, selected at random, was more likely to be a nurse than a surgeon, based on solely current statistics ("The nursing and midwifery workforce is 91% female compared to 9% male"; "The ratio of male to female consultant surgeons in the UK is approximately 8:1" — statistics from the Royal College of Nursing and the Royal College of Surgeons respectively).
Jarek Weckwerth said,
June 27, 2024 @ 12:05 pm
@ Philip Taylor worries me [that] it would be considered inappropriate You're a man of the world, Philip. In "polite society", there have always been facts (!) considered inappropriate for public discussion. What exact facts those would be has changed over the centuries, but the general principle stands. Today, the statistics of some social features is one such topic.
RfP said,
June 27, 2024 @ 2:08 pm
At the risk of being cryptic, I’ll just leave it at this:
“Plus c’est la même chose, plus c'est la même chose.”
Anubis Bard said,
June 27, 2024 @ 2:08 pm
To answer your question @Philip Taylor – I can perfectly well regard your assumption as inappropriate because – statistical facts aside – the assumption so often helps to normalize and perpetuate the legacy of a sexist past. The author points out that LLMs don't have any way to really grok these arbitrary cultural projects or conventions (like eroding that legacy, or putting the statistical minority on equal footing) versus basic statistical fact, but I think people ought to.
Gregory Kusnick said,
June 27, 2024 @ 7:32 pm
Statistics can tell you something about what fraction of women at parties are nurses. The mistake is assuming they can tell you anything useful about the profession of a specific woman at a specific party. In the individual case, probability is just a measure of your own ignorance. So the appropriate thing to do is to remedy your ignorance before making any assumptions.
John Swindle said,
June 27, 2024 @ 9:10 pm
"To assert that someone is a nurse and that they do not work in healthcare is a contradiction": Unless of course we accept older definitions of "nurse."
"… at the present time, … in large training samples the word nurse occurs far more frequently in the context of hospital than of theater, an observation grounded in its meaning." This is surely true, although the occasional occurrence of the word "nurse" in the context of "operating theater"/"operating theatre" would also be grounded in its meaning.
My late foster daughter was deaf and blind from birth or near birth and communicated by a version of tactile Signed English. When told that an aunt had graduated from med school and become a physician, she turned to us urgently and signed "Nurse?"
Jarek Weckwerth said,
June 28, 2024 @ 2:58 am
@ Gregory Kusnick: In the individual case, probability is just a measure of your own ignorance. So the appropriate thing to do is to remedy your ignorance before making any assumptions.
Typically, an LLM will not have any way of remedying its ignorance, mostly because it is not dealing with a specific case, so the ignorance itself is a probability. Thus the only reasonable thing to do is to fall back on statistics; otherwise you end up with a black Nazi soldier fiasco a la Gemini.
This is done all the time in advanced societies where you need to deal with large numbers of people you cannot be expected to know. The airline seat is sized the way it is because that is reasonable statistically. And it is generally accepted that you cannot accommodate extreme minority sizes in reasonable ways, as sad and disappointing as it may be.
Philip Taylor said,
June 28, 2024 @ 3:31 am
OK, let me try a simpler approach. Could we all agree that at a party at which only surgeons and nurses are present, the majority of the men will probably be surgeons and the majority of women will probably be nurses (statistics as cited in previous comment) ? If so, then is it not reasonable to assume that a random woman at the party is probably a nurse, and a random man at the party is probably a surgeon ? These assumptions are clearly borne out by the statistics, so why do some find them contentious ?
John Swindle said,
June 28, 2024 @ 6:30 am
The author's point is that the biases of large language models when they work correctly are those of the users of the language. It's not about how many surgeons or nurses or males or females are at the party. It's about how many surgeons or nurses or males or females the users of the language think would be at the party. It works just as well for attributing popularly-assigned characteristics to gays or immigrants or infidels or Jews.
Philip Taylor said,
June 28, 2024 @ 7:01 am
I do not dispute for one second, John, that if an LLM is reflecting bias on the part of those who have contributed to its source data, then its outputs are open to challenge on the basis that they are reflecting large-sample bias. But that is not the point that I was seeking to challenge. Philip Resnik writes (in part)
and I maintain that that statement is wrong. It is not "normatively unacceptable" to assume that, in a gathering of people all of whom are drawn from the medical profession, the women are more likely to be nurses than doctors or surgeons, and the men are more likely to be doctors or surgeons than nurses. It is totally acceptable, because such an assumption is clearly borne out by the statistics.
In just the same way, if random police checks reveal that the probability of finding illegal drugs or weapons in a car driven by members of group X is significantly greater than the probability of finding similar items in cars driven by members of groups Y1, Y2, … Yn, then the police are perfectly justified in stopping vehicles driven by members of group X more frequently than they stop vehicles driven by group Y1, Y2 … Yn, and indeed I would want/expect them to stop such vehicles more frequently than others so as to reduce the amount of drugs / number of illegal weapons in circulation.
Andrew Usher said,
June 28, 2024 @ 7:46 am
I think many would agree there, but not all, and presumably not Resnik. Whether using certain prejudices is acceptable is not simply a matter of fact. However, your example is a rather contrived situation and it's perhaps not easy enough to see how that type of decision would make a difference in the real world. In that area I think one could talk about a normal mecial visit: one may assume an unknown person approaching is a/the doctor or nurse depending on gender; and I'm sure female doctors and male nurses are used to being mistaken as the other by patients and simply have to deal with it.
It would have been reasonable, I think, for the author, Philip Resnik, to have included some example of how these biases could be allegedly harmful in real use of LLMs – many readers might not be acquainted with anything beyond, perhaps, getting ChatGPT to say some possibly offensive things, which is not serious. I don't see how this is an LLM problem either – if they are only as biased as the average person, how does that change the total bias in society? And he dismissed, certainly, too cavailierly the retort that 'people are biased too' – yes, social pressure can cause people to hide their biases, but not directly to eliminate them. If LLMs pick up on that, they're being smart in exactly the way we want them to be, and as he almost concedes, that necessarily means they'll expose things people don't like to think about.
And that leads into my next point. Jarek Weckworth said:
> You're a man of the world, Philip. In "polite society", there have always been facts (!) considered inappropriate for public discussion.
No doubt he's aware of politeness conventions: there are things, not limited to those obscene, not generally appropriate in public discussion. But this surely does not go far enough. Stipulating these things to be fact, as you say, it is not just that mentioning these facts is considered impolite. No, we have people _denying_ them, in public, at almost every opportunity, and that's usually not considered impolite. And when someone does express one of these 'impolite' facts, it's more than just a faux pas, but treated as cause for righteous condemnation. The best one could call this is hypocrisy (something that, in my opinion, has never been of benefit), not politeness.
The same kind of thinking that gives rise to hypocrisy on the social level also generates all other cases of people not talking about, or thinking about, something that needs to be talked and thought about, and that broadly is the cause of all the soluble problems, evils, and real injustice in society.
k_over_hbarc at yahoo.com
Benjamin E. Orsatti said,
June 28, 2024 @ 8:42 am
Maybe we're failing to make a distinction here: There are innocuous ways through which life experience leads us to make generalizations and predictions about future events (heuristics). 100% of the time I've called a particular Chinese restaurant to place a take-out order, I've been met with a somewhat-dysfluent, Chinese-accented, voice on the other end of the line. So, when I'm dialing the number, some part of my consciousness is "priming" my speech centers to generate slow diction and simple vocabulary so that I end up getting my delicious tripe, instead of, say, something with hooves in it, which is important to me. In the event my interlocutor greets me with perfectly-fluent, unaccented Western Pennsylvania English, I adapt my prediction to conform with these "new facts" and speak as I normally do.
But what if I'm an AI-HR director bot of a retail store that happens to "know" that x% of internal retail theft in the jurisdiction is committed by women, and y>x% is committed by men? I am programmed to "screen" job applicants and winnow them down to, say, 5, and one of the primary "qualifications" is "likelihood to steal from the store". All 5 of the job applicants are going to be women, and the AI-bot isn't going to necessarily have the opportunity, as would a human, to "pivot" and adjust to new information.
So, one of the reasons that AI should never be used at all by anyone ever, but rather should be fled from like a demon from the pits of hell is that people are going to clumsily use AI to make decisions that actually effect people's lives (e.g., their livelihoods) because it's the easiest thing to do.
People are marvelously plastic, but a computer will do _exactly_ what you tell it to do.
Stephen Goranson said,
June 28, 2024 @ 9:57 am
AI assigned to me a publication which I not only did not write (a different Steve did), but one that I had criticized online.
Then I saw a headline on possible side effects of turmeric. News to me. Because it was paywalled, I did a search. First hit:
Amazon offering "best deals on turmeric side effects"!
Gregory Kusnick said,
June 28, 2024 @ 10:20 am
Phillip, here's where "statistically defensible" departs from "normatively acceptable" for me:
Suppose I'm at that party of surgeons and nurses, working my way through the crowd chatting with each guest in turn. If I approach each woman with the idea that she's probably a nurse, sooner or later that attitude is going to get me in trouble, particularly as blood alcohol levels increase. The more responsible approach is to realize that I don't know what anybody's profession is until I ask them, even if "probably a nurse" might be statistically defensible. ("Might be" because if I make it to the other end of the room having decided that every single woman is"probably a nurse", then it seems like I've made an error somewhere.)
Philip Taylor said,
June 28, 2024 @ 10:31 am
I agree, Gregory. Just because the majority of women in the room are (on the basis of valid statistics) probably nurses, that does not imply that one can assume that any one woman is a nurse, any more than one can assume that any one man is a surgeon or doctor. Statistics tell us about probabilities over a given population, nothing at all about a single member of that population.
/df said,
June 28, 2024 @ 6:15 pm
How shocking that artificial intelligences (however limited, discuss) designed to learn like the human brain should acquire similar common sense "biases" from the same real world data!
Philip Resnik said,
June 28, 2024 @ 8:36 pm
I wanted to express my appreciation to Mark for the post and to all of you on this thread for your comments. I'm keeping track of these and if with luck the article is accepted, I will be reviewing what folks said for additional insights, beyond the reviewers', that can inform revisions. One of those would certainly be greater clarity, which Philip Taylor's comment motivates, regarding the relationship between statistical inferences and what may nor may not be normatively acceptable. I agree with a number of the counterpoints, and I think we all (including Philip Taylor) have agreed it is normatively unacceptable to draw conclusions, based on statistics, about what *is true* of an individual. Philip T's point about what one can conclude *is likely* is, I agree with him, at minimum a subtler point.
Although I have not yet finished thinking about this, my thoughts are tending toward the following reasoning that conclusions about *likelihood* (even beyond the perpetuation of bias that Anubis Bard and others pointed out) can still, I think, be normatively questionable. A bank's decision on whom to give a loan, for example, is largely about the likelihood of default. Even if defaulting were statistically more likely for person of a particular defined group (e.g. a member of a protected class, in the U.S. sense of that term), I am pretty sure it would not be considered acceptable for the bank to deny the loan solely on that basis. To take a stab at expressing this a bit more precisely, the conditional probability we're talking about is Pr(X is true of person P | G). In some cases (e.g. when X is "is a nurse" and G, for "group", is "person wears blue to work"), that's a conditional probability it's ok to use. In others, though — and here's my attempt at refinement — this may be *marginally* true, in the sense of marginal probability: person P may have lots of other properties, say A, B, and C, so the full joint probability distribution is Pr(X is true of person P | G, A, B, C). But marginalizing out A, B, and C to condition only on G amounts to disregarding those properties, calculating the probability being used *only* on the basis of the group G property. I think (though again, caveat, I haven't fully thought it through) that that's where the normativity issue shows up even for conclusions that are valid statistically. Even when talking about the *probability* that X is true of an individual, marginalizing out A, B, and C is effectively taking a position that G is the *only* property that matters, that needs to be paid attention to. That's normatively ok for some groups ("wears blue"), but not for others.
Jarek Weckwerth said,
June 29, 2024 @ 4:10 am
@Philip Resnik: Thank you for this post. This is the kind of thing that makes Language Log so great!
That's normatively ok for some groups ("wears blue"), but not for others. I think this is what Philip Taylor was getting at in his first post (but of course he can disagree). Why exactly is it OK for some groups not others?
Also, I think there is the implication that, for some groups, this is a "newly acquired power/protection", and my point was exactly that those groups/sets of facts have always been there but they change over time.
An additional factor may be the use of the word bias which implies a lack of fairness etc., where a less loaded* term like skew would do just as well.
Anyway, it seems to me that these kinds of "taboos" simply cannot be learned from the data alone (either language data, like in the case of LLMs, or any other type of data in general, e.g. visual data). Humans learn these things from acculturation, and for the time being, it seems like this needs to be hard-coded in one way or another, as in the case of the hidden Gemini prompts.
And finally, I'm not sure if the bank loan analogy (or the party analogy, for that matter) is a good one for the nurse situation we discussed earlier. If you ask Dall-E or Gemini to make a picture of a nurse, it has to make a binary (sorry!) decision, male or female. If you tell it to ignore the real-world probabilities, then you must specify that it is only that feature that needs to be ignored. Otherwise, you will be getting pictures of small boys in diving suits on the tail fin of a airliner, and the users will be giving them thumbs down.
In the bank loan case, a good system would probably have in its training data examples of other rating factors beyond simple group membership (e.g. from existing loan application forms or descriptions of credit rating procedures). In other words, applying only group membership would not in fact reflect real-world data the way the statistics of nurse gender do.
(*) Not a "native speaker", so correct me if my intuition is wrong.
Philip Taylor said,
June 29, 2024 @ 5:18 am
Jarek — "Humans learn these things[taboos] from acculturation, and for the time being, it seems like this needs to be hard-coded in one way or another, as in the case of the hidden Gemini prompts" — yes, "[h]umans learn these things from acculturation", but they are able to decide for themselves whether or not (and when) to respect the taboo — you will see from one of my more recent comments that, when I feel the situation warrants it, I am willing to ignore the n-word taboo (a situation in which we are discussing the word itself, rather than the group of people that it pejoratively denotes). So simply hard-coding these taboos into LLMs is not the right approach, IMHO; rather, the LLM should have access to the current set of taboos, and flag up when it is breaching one of these. After all, the OED itself lists the n-word in full, but goes on to say "This word is one of the most controversial in English, and is liable to be considered offensive or taboo in almost all contexts (even when used as a self-description)". It would be a very sad day indeed (IMHO) if the OED had to list the word as "N****r" or similar.
Seth said,
June 29, 2024 @ 5:35 am
There's deep problems here, of the conflict between the results of fairly low-level algorithms versus desired high-level responses. It's hard to discuss these issues, since they are indeed very political.
In brief: Assume nurse/doctor is more female/male. Ask for "A picture of a nurse (doctor)". What SHOULD the response be? A simple algorithm is to use a "most probable" result. But this would mean nurse ALWAYS female, doctor ALWAYS male. That seems very wrong. First attempt at patch, generate a random female/male according to the proportion. First problem, what is the proportion to use? It's very different depending on age, specialty, and so on. But pick something anyway. Now what happens with "A picture of nurses (doctors)". Simply doing random female/male will eventually generate just by a chance an image of all female nurse and all male doctor. This will be a scandal. Moreover, it's arguably not what the user wanted. Thus there needs to be some sort of rule that any group should also have proportion.
However, some groups are intrinsically disproportionate. Otherwise it leads to Google Gemini kinds of problems with Asian women Popes and black women Roman Emperors.
And that's at the most elementary level. Now factor in the contentious politics around representation and identity, and this is all extremely hard.
Andrew Usher said,
June 29, 2024 @ 7:04 am
Benjamin Orsatti wrote:
> But what if I'm an AI-HR director bot of a retail store …
> So, one of the reasons that AI should never be used at all by anyone ever, but rather should be fled from like a demon from the pits of hell is that people are going to clumsily use AI to make decisions that actually effect [sic] people's lives …
I can assume that statement was not entirely serious, but even so, it makes a point. Should we be afraid of AI? I try to judge new technology fairly, seeing both good and bad, and the new AI so far has not shown much good. But in this field I can see nothing to be afraid of. Companies already do, or have done, all kinds of nasty things in the hiring process, and I can't see how AI makes it qualitatively worse, for the fact that the decision 'affects people's lives' and is subject to bias already exists. Such a bias as you give here as an example would surely be screened out before putting the system into service; it's both easy and legally advisable to test it for biases – while human decision-makers are not. And one can be sure that, from its nature, any remaining biases it has have some rational justification, unlike those of people.
Indeed, human biases (conscious or, more likely, not) are so large a part of hiring _now_ that it is absurd to ignore them in any discussion of future biases. The largest and most pervasive human bias is that human judgement is required in the first place, while in fact human judgement of strangers is known to be bad (especially when that person is trying to influence it – a skill, but hardly a redeeming feature, nor associated with others as far as I know). I have long believed that for most of the workforce, hiring based on an algorithm rather than human judgement would give better results for both sides, and AI is just another kind of algorithm here.
The pervasiveness of prejudice and stereotypes throughout history is often taken as some sort of essential moral flaw, but it's more likely to be making up for this deficit in judgement – no matter the actual nature of the 'other' group, a prejudice against them will reduce the chance of getting scammed.
John Swindle said,
June 29, 2024 @ 7:16 am
In a family, would we want children to take out the trash if the adults could probably do it better? Can we agree that the adults probably could do it better? And that in a specific case the child might do it better? And that there might be other considerations?
We're at the intersection of ethics, language, and new technology. We can do a lot of things with language and a lot of things with technology. We see every day that language that imitates human language looks like human language. As /df says, how shocking. I guess we don't yet have a machine or even a language that will take over the burden of being kind or fair for us.
John Swindle said,
June 29, 2024 @ 8:27 am
Philip Taylor, thanks. It's important to get the probabilities right, but I'm afraid that won't be enough to solve the ethical problems.
Philip Taylor said,
June 29, 2024 @ 8:41 am
But should AI/LLMs seek to make ethical judgements, John ? If the human race could agree on a single code of ethics, then I would say "yes", but all the while that what is ethical is open for debate[*], then I would argue "no" — ethical judgements must be left to those using the outputs of AI/LLMs, not trusted to to the system(s) generating those outputs.
Examples : is the slaughter of animals in accordance with Halal requirements ethical ? Is plunging a live lobster into boiling water ethical ? Is it ethical to eat a fertilised hen's egg that is close to hatching ? Can war be ethical ?
Jonathan Smith said,
June 29, 2024 @ 11:31 am
While I guess it's counterintuitive, it seems clear that pictures of "Asian women Popes and black women Roman Emperors" (quoting Seth) are exactly what such tools *should* be generating — the jarring wrong-seemingness of such images is OUR problem. As the user, your supposed to be like "oo nice but shorter hair / lighter skin / whatever please."
Philip Taylor said,
June 29, 2024 @ 11:37 am
"OUR problem", Jonathan ? "the jarring wrong-seemingness of such images" is because they are wrong (where "wrong" = "completely unattested"). There are (and have never been) any Asian woman Popes (and I am certain that there never will never) and there never were any black women Roman Emperors.
Benjamin E. Orsatti said,
June 29, 2024 @ 11:48 am
Andrew Usher said, [AI/LLMs have biases, people have biases, and we can “screen” for them in both cases, so take off your tin foil hat and relax a little, big guy]. (I paraphrase).
I don’t know enough about programming to speak to the extent to which you can “screen” for such things (I’m just barely competent at Microsoft Excel, and did a little happy dance when someone on LL showed me how to html a block quote). Maybe the EEOC will publish Regulations governing AI, and that’ll be a “good enough” fix insofar as hiring decisions go.
But it’s the _other_ thing. Writing changed the human brain. A written culture is different from an oral culture. Going from scrolls to codices (i.e. “books” changed the way people read, studied, and thought, beginning in late antiquity. The printing press democratized (sorta) and accelerated this development. Inter alia, our memories ain’t what they used to be. Now, the internet is here. Now AI is here.
I have access to boolean-searchable databases of laws, jurisprudence, and treatises, and can collect information in an hour that would have heretofore taken a day spent in a law library. In a common-law legal system reliant on caselaw and administrative interpretation, this is just delightful.
…but lately, Westlaw is pushing all this AI garbage, uh, I mean “tools” on me, and I don’t wanna. I want to use my own soggy gray meatloaf to apply facts to law and law to facts; not rely on some ticker tape spewed forth from the machine. What does the machine know of Constitutional principles? We are trained to “think like lawyers”. When we delegate this, we will “learn” to think like the machines, to the extent human reasoning enters the picture at all.
Let’s end with a shudder, shall we? It’s not only lawyers who are letting AI do their thinking for them: with federal courts in particular whinging about oppressive caseloads, what happens when (if it has not already happened) it’s the AI bot who’s really banging the gavel?