Everyday statistical reasoning
« previous post | next post »
Mark Liberman has been writing about the persistent misinterpretation of claims about statistical differences between groups with respect to some property — misinterpretations in which these differences (often small ones) are understood as general (and essential) differences between the groups. A little while ago, Mark suggested that reporting the differences by means of generic plurals ("Asians have a more collectivist mentality than Europeans do") promotes misunderstanding and proposed that such generic language be avoided.
In comments, Mark noted that changing the language of popular science reporting is not by itself going to fix an inclination to essentialist thinking (which is all over the place), though it might be a step in the right direction.
Mark looked at groups X and Y with respect to property P, focussing on statistical differences between X and Y. Let's say that members of X tend to have P to a greater degree than members of Y. Then there's a statistical association (perhaps a very weak one) between X and P. And most people have as much trouble understanding statistical association as they do statistical differences. Here, the characteristic error in reasoning is treating the association as invariant: all Xs have P (and no non-Xs do). Again, the error is all over the place, often showing up in objections to claims of statistical association. As in the following case from a September 30 NYT story "Dementia Risk Seen in Players In N.F.L. Study" by Alan Schwarz.
The story begins with a report of an apparent statistical association:
A study commissioned by the National Football League reports that Alzheimer's disease or similar memory-related diseases appear to have been diagnosed in the league's former players vastly more often than in the national population — including a rate of 19 times the normal rate for men ages 30 through 49.
Now, the study has a variety of shortcomings, and the association can't be taken as demonstrated. but for the moment let's take things at face value.
What especially interests me in the story (though I have a long-standing interest in dementia and related conditions) is the reaction of NFL spokesman Greg Aiello, who
said in an e-mail message that the study did not diagnose dementia, that it was subject to shortcomings of telephone surveys and that "there are thousands of retired players who do not have memory problems."
"Memory disorders affect many people who never played football or other sports," Mr. Aiello said.
Aiello's response addresses (and rejects) claims not made in the report: that all retired players have memory problems (all Xs have P), and that no other people have memory problems (no non-Xs have P). That is, Aiello chose to treat the association as invariant, when that was clearly not what the report said. I don't see how the reporter could have averted this misunderstanding by more careful wording.
Such misunderstandings — occasioning objections much like Aiello's — arise even when claims are not couched as statistical associations. Quantified statements like "many Xs have P" will do, eliciting objections like "but I'm an X and don't have P". People reason like this all the time, and that's why there are Critical Thinking courses.