Most people don't understand statistical ideas like distribution, correlation, and regression. And even if they understand these concepts, they often find them too complicated for everyday thinking, or impractical for everyday communication. So instead they fall back on essentialist beliefs and generic statements.
For example, "The lower classes are not merely unfortunate, according to the upper classes; they are genetically inferior". This statement attributes a belief about one generic category ("the lower classes") to another generic category ("the upper classes"). It might be someone's subjective assessment based on personal experience, but in this case it's a journalist's description of the results of a series of psychological experiments, describing the fact there was a significant correlation (and a significant multiple-regression coefficient) between a "social class rank" variable and an "essentialist beliefs about class" variable. And as generic statements often do, it risks leaving readers with misleading ideas about the experiment and about the attitudes that it tested.
Here's the context for that quotation — Matthew Hutson, "Social Dawinism Isn't Dead", Slate 1/3/2014:
In 2012 the top 0.01 percent of households earned an average of $10.25 million, while the mean household income for the country overall was $51,000. Are top earners 200 times as smart as the rest of the field? Doubtful. Do they have the capacity to work 200 times more hours in the week? Even more doubtful. Many forces out of their control, including sheer luck, are at play.
But say you’re in that top 0.01 percent—or even the top 50 percent. Would you want to admit happenstance as a benefactor? Wouldn’t you rather believe that you earned your wealth, that you truly deserve it? Wouldn’t you like to think that any resources you inherited are rightfully yours, as the descendant of fundamentally exceptional people? Of course you would. New research indicates that in order to justify your lifestyle, you might even adjust your ideas about the power of genes. The lower classes are not merely unfortunate, according to the upper classes; they are genetically inferior.
The research in question was reported in a paper by Michael Kraus and Dacher Keltner, "Social Class Rank, Essentialism, and Punitive Judgment", Journal of Personality and Social Psychology, 2013. Their abstract:
Recent evidence suggests that perceptions of social class rank influence a variety of social cognitive tendencies, from patterns of causal attribution to moral judgment. In the present studies we tested the hypotheses that upper-class rank individuals would be more likely to endorse essentialist lay theories of social class categories (i.e., that social class is founded in genetically based, biological differences) than would lower-class rank individuals and that these beliefs would decrease support for restorative justice—which seeks to rehabilitate offenders, rather than punish unlawful action. Across studies, higher social class rank was associated with increased essentialism of social class categories (Studies 1, 2, and 4) and decreased support for restorative justice (Study 4). Moreover, manipulated essentialist beliefs decreased preferences for restorative justice (Study 3), and the association between social class rank and class-based essentialist theories was explained by the tendency to endorse beliefs in a just world (Study 2). Implications for how class-based essentialist beliefs potentially constrain social opportunity and mobility are discussed.
That all makes such perfect sense that some might see this research as analogous to experimental proof of the wetness of water. But that reaction would be inappropriate, since the most striking thing about the experimental results, in my opinion, is how weak they were: the tested fluid is not really very wet at all.
I'm afraid that the facts of the case are complicated, and so it's going to take a while to go through (even some of) them. If you're pressed for time, or not especially interested in the hard life of science writers (and Matt Hutson is a good one, in my opinion), you might want to turn your attention to some of our shorter posts.
Let's start with some of the numbers, and then see what they mean. Here's a table of correlations from that Kraus & Keltner paper:
In order to understand this table, we need to know what the authors mean by the ten variable names that label the rows and columns.
Variable (1), "Social class rank", is the result of a guided self-assessment. Participants "[were] asked to rate their social class based on a measure of subjective socioeconomic status used in previous research (Adler et al., 2000; Kraus et al., 2009)". The cited reference is N.E. Adler et al., "Relationship of subjective and objective social class with psychological functioning: Preliminary data in healthy white women", Health Psychology 2000, which explains that
Participants were given a drawing of a ladder with 10 rungs that was described as follows: “Think of this ladder as representing where people stand in our society. At the top of the ladder are the people who are the best off, those who have the most money, most education, and best jobs. At the bottom are the people who are the worst off, those who have the least money, least education, and worst jobs or no job.” They were then asked to place an X on the rung that best represents where they think they stand on the ladder.
Variables (2) and (3), "Income" and "Education" were measured through simple multiple-choice questions:
Educational attainment was assessed with four categories: (a) less than high school education, (b) high school education, (c) college graduation, and (d) postgraduate degree. Annual income was assessed with eight categories: (a) less than $15,000, (b) $15,001–$25,000, (c) $25,001–$35,000, (d) $35,000–$50,000, (e) $50,001–$75,000, (f) $75,001–$100,000, (g) $100,001–$150,000, and (h) greater than $150,000.
Note in passing that the highest-income category (h) corresponded to about the top 10% of Americans in 2011. And also note that although Social class rank is significantly correlated with Income and Education, the correlations are not very strong: r=0.36 and r=0.21 respectively. We can square these correlations to get "percent of variance accounted for", which tells us that the self-reported Social class rank measure accounted for about 13% of the variation in Income, and about 4% of the variation in Education. To get a sense of what a correlation of r=0.36 means, here's a plot using fake data (unfortunately Kraus & Keltner don't provide the underlying data tables from their experiment):
Variables (4), (5), and (6) ("Biological beliefs", "Discreteness beliefs", and "Informativeness beliefs") were determined by averaging Likert-scale responses to a set of questions about each topic, e.g. "the kind of person someone is can be largely attributed to their genetic inheritance", "everyone is either a certain type of person or they are not", "it is possible to know about many aspects of a person once you become familiar with a few of their basic traits". There were eight questions for the first two belief scales and seven for the last one.
How about the measure of Variable (7), "Social class essentialism"? The authors used standard psychometric techniques to find ten questions whose answers can be combined to yield one overall measure and two sub-scales that correspond roughly to their preconceptions about such attitudes:
On the basis of previously developed self-report measures of essentialism, we created 22 items for assessing people’s essentialist beliefs about social class categories. Essentialist conceptions of social class include beliefs that it is easy to judge social class through brief interaction, that children can be separated into different social class categories at birth, that social class can be determined without clothing cues, and that social class is at least partially based in biological temperament or genetic tendency. We administered these 22 items to a large sample (n = 703) of undergraduate students at a public West Coast university. The sample was collected as part of a larger battery of psychological questions administered to psychology majors online, at the start of the academic semester.
We then subjected participant ratings to a principal-components factor analysis with a varimax rotation. Examination of the scree plot revealed a first factor with an eigenvalue of 5.30 and a second factor with an eigenvalue of 2.05. [...] When we inspected the items, the first factor appeared to tap into beliefs that social class is a discrete social category (e.g., “Other people’s social class is easy to figure out”). The second factor appeared to tap into beliefs that social class is biological (e.g., “Social class is partly biological”). We selected items for the two factors of the essentialist beliefs about social class categories scale if the item loaded at .40 or higher on one factor and .30 or below on the other factor. This procedure yielded a total of 10 items: six on the discreteness subscale (M = 3.93, SD = 0.96) and four on the biological subscale (M = 2.64, SD = 1.00). The two scales were significantly positively correlated (r = .35, p < .05), and so a total score for essentialist beliefs about social class categories was computed (M = 3.42, SD = 0.81, α = .74).
The variable they call "Social class essentialism" then corresponds to that "total score for essentialist beliefs about social class categories"; and the "Discreteness subscale" and the "Biological basis subscale" (variables (9) and (10)) are the two PCA dimensions that they extracted from the pattern of answers to the ten questions.
Variable (8), "Conservatism", was based on direct self-assessment:
To assess political orientation, we collected a single item assessing the extent to which people self-identify as liberal or conservative on a 7-point Likert scale (1 = very liberal, 7 = very conservative; M = 3.63, SD = 1.67).
Finally, who were the subjects?
Participants were 169 adults from a national online sample participating for the opportunity to win small gift certificates to an online retailer. The participants were recruited through advertisements on http://Craigslist.org, and the website where data were collected was maintained by a public West Coast university. Of the 169 participants (age M = 35.04 years), the majority (n = 112) were female. Participants self-identified primarily as European American (n = 112), Asian American (n = 27), other or multiple ethnic groups (n = 17), African American (n = 9), Native American (n = 3), or Latino (n = 1).1 The social class backgrounds of participants in the sample were also quite diverse: 42.9% of participants reported annual incomes of $50,000 or less, and 39.3% of participants reported graduated from high school as their highest level of education completed.
So how about the rest of the results?
From their Table 2 (as reproduced above), we can see that there was indeed a positive and statistically significant correlation (of r=0.24) between their Social class rank and Social class essentialism measures. The correlation between Conservatism and Social class essentialism (r=0.17) was also statistically significant, though smaller. But "statistically significant" here just means that the observed numbers are probably not simply due to sampling error in comparing two populations whose distributions are in fact identical. If we ask how useful e.g. self-reported "Social class rank" is in predicting their measure of "Social class essentialism", the answer is that it accounts for 0.24^2 = .0576 or about 6% of the variance.
Again in graphical form, here's what a correlation of r=0.24 looks like (and again this is fake data — at some point in the marvelously improved future, authors of scientific papers will routinely be required to supply their tables of raw data in digital form):
In fairness to Kraus & Keltner, a correlation of r=0.24 is quite good by the standards of social psychology. Here's a plot of "results from a century of social psychological research, more than 25,000 studies of 8 million people", reported in F.D. Richard, C.F. Bond, and J.J. Stokes-Zoota, "One hundred years of social psychology quantitatively described", Review of General Psychology, 2003, and previously discussed in "The and a sex: a replication", 8/16/2009.
And small but real effects can sometimes be practically important: in elections, for example, a percent or two of the vote is a big deal.
Returning to Kraus & Keltner, the correlation between Income and Social class essentialism (0.12) was not statistically significant, undermining even the statistical version of Matt Hutson's framing of social class essentialism as an attitude naturally characteristic of "that top 0.01 percent [of the income distribution]—or even the top 50 percent".
This all naturally leads us to ask why K&K to be having such a hard time showing that water is wet. Perhaps the assessment methodology is inherently noisy. But it's possible that the fluid in question just isn't in fact very watery, apparent common sense to the contrary. After all, Americans are raised on a diet of Horatio Alger stories, which presuppose both meritocratic possibilities and non-meritocratic realities, and we're often taught that hereditary aristocracy is a bad idea that our ancestors emigrated to the America to escape. There are other factors that tend in the opposite direction, especially racial and ethnic stereotypes, but it shouldn't be a shock to discover that Americans, whatever their income and self-assessed class standing, might be resistant to baldly-stated "social class essentialism".
It's also possible that the relevant space of social attitudes involves some complexities that K&K glossed over. In this connection I'd like to point to an interesting fact about their Table 2: the observed correlation between Income and Conservatism is 0.00.
Compare this to the work of Andrew Gelman, Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do, 2009 (powerpoint summary here). Stealing a couple of his slides:
He resolves the apparent paradox this way: "Context matters [...]: The poor are similar in Red and Blue America, but the rich are different."
The ideas underlying "Social class essentialism" are (I think) a mixture of those generally described as "social issues" and those generally described as "economic issues"; and Kraus & Keltner's sample of 169 subjects is some mixture of the groups underlying Gelman's distinction between "Republican States" and "Democratic States". So a larger and better-described sample might show us a clearer picture with stronger relationships between social categories and attitudes.
Whatever the reasons for the low correlations in K&K's study, we're left with a typical difficulty for the interpretation of generic statements as a summary of weak statistical tendencies. As I wrote in "Generic comparisons", 11/7/2011:
[T]he relationship between mere prevalence and … generic statements is a very weak one. Everyone knows that "Ticks carry Lyme Disease", although only a minority of ticks do so (14% in one study). Everyone knows that "Mosquitoes carry West Nile Virus", though the highest infection rate found in the epicenter of a recent epidemic was estimated at 3.55 per thousand (and the rate was essentially zero outside of the epicenter).
As a result, I once suggested banning such statements altogether ("Mandatory treatment for generic plurals", 9/13/2009):
I propose a voluntary ban on the use of generic plurals to express statistical differences, especially in talking to the general public about scientific results in areas with public policy implications. In other words, when we're looking at some property P of two groups X and Y, and a study shows that the distribution of P in X is different from the distribution of P in Y to an extent that is unlikely to be entirely the result of chance, we should avoid explaining this to the general public by saying "X's have more P than Y's", or "X's and Y's differ in P", or any other form of expression that uses generic plurals to describe a generic difference.
But this is completely impractical. Like all other natural languages, English offers us the irresistable temptation to substitute simple generalizations for complex ideas about the complex relationships implicit in tables of observations. You might as well try to ban alcohol or marijuana.
So instead, let me suggest a policy of responsible indulgence. Let's focus on preventing people from driving public policy debates when their judgment is impaired by generic intoxication, and offer effective statistico-medical intervention to those whose addiction to essentialist thinking blights their consciousness in personally as well as socially destructive ways.
Update — It's not just science writers who are tempted to make careless generic statements. Researchers themselves are often equally guilty. Thus Joshua Holland, "Why the Wealthy Favor Harsh Punishment — for Criminals and Errant Schoolchildren", Moyers and Company 1/13/2014, quotes Dacher Keltner as saying:
We asked people from different class backgrounds to think about the rich and the poor. And then we asked them, “To what extent do you think that these categories, rich and poor, are about people who have different genes, or different temperaments, or different biological makeups?” And again we found this similar pattern, which is that upper class individuals think of class as being based in biology and genes, and you don’t see that belief in people who are less wealthy.
He says that "upper class individuals think of class as being based in biology", not "upper class individuals tend to think of class as being based in biology". Fair enough — so far this is just a typical generic statement like "Ticks carry Lyme Disease". But then he tells us that "you don’t see that belief in people who are less wealthy". This statement must be technically false, given the correlation of merely r=0.20 between the Income variable and the Biological beliefs and Biological beliefs subscale variables in their study.