According to The Economist, 4/2/2009, "Neuroscience and social deprivation: I am just a poor boy though my story's seldom told":
THAT the children of the poor underachieve in later life, and thus remain poor themselves, is one of the enduring problems of society. Sociologists have studied and described it. Socialists have tried to abolish it by dictatorship and central planning. Liberals have preferred democracy and opportunity. But nobody has truly understood what causes it. Until, perhaps, now.
The crucial breakthrough was made three years ago, when Martha Farah of the University of Pennsylvania showed that the working memories of children who have been raised in poverty have smaller capacities than those of middle-class children. [emphasis added]
OK, keep that bold-faced statement in mind. Now let me offer you a bet. Suppose we have a large group of "children who have been raised in poverty", and another large group of "middle-class children", taken from the same groups described in the cited research; and we measure the capacity of their working memories, using the same testing techniques as the cited research. We pick one of the "children raised in poverty" at random, and one of the "middle-class children" at random. Will you bet me that the rich kid will outscore the poor kid, giving odds of, oh, say, 2-to-1? (That is, you put up $200 and I put up $100.)
You might not like to gamble on single events — I certainly don't — so let's give you the benefit of the law of large numbers. We'll run this same bet 100 times, with 100 different random pairs of kids. Will you take the challenge now?
It certainly seems like you should — if "the working memories of children who have been raised in poverty have smaller capacities than those of middle-class children" is a statement about all poor kids and all rich kids, then this ought to be a quick way for you to pick up $10,000, right?
But if you're a sensible person, then your response will be that you have no idea from the Economist article whether I'm offering you a sure thing, a fair bet, or a probable loss. And I have an advantage over you, which is that I know the details of the research that the Economist is talking about.
The reference is Martha J. Farah et al., "Childhood poverty: Specific associations with neurocognitive development", Brain Research 1110(1):166–174, 2006. Martha and her colleagues gave a range of tests to middle SES and low SES African-American girls between the ages of 10 and 13. As I pointed out in a blog post a couple of years ago ("Cultural specificity and universal values", 12/22/2006), the biggest effect found in this important study was actually in in language-related cognitive measures — an effect size of about d=0.95 for vocabulary and sentence-understanding tests:
The second-biggest effect, of about d=0.65, was for what they called "the Medial temporal/Memory composite", which involved "incidental learning tasks" (I think that this involved the unattended background of another task.) The Economist's article focuses on what Farah et al. call "the Lateral prefrontal/Working memory composite" — a combination of a spatial working-memory task (something like "where in that array of numbers was the 3?") and a two-back task ("press the button when the stimulus is the same as the one before the previous one") — which had an effect size of about 0.5.
What does an "effect size of 0.5″ mean about the distribution of performance among the middle and low SES subjects? This means that the average value for the first group was about half a standard deviation higher than the average value for the second group. If the distribution of values within each group was "normal" (i.e. gaussian), then idealized forms of the two probability density functions would look like this:
(For an example of what empirical distributions look like in such comparisons, see this blog post.)
Another way to think about such overlapping distributions is to ask how often a randomly-selected member of one group would outscore a randomly-selected member of the other group. In this case, we'd expect the random middle-SES girl to do better on a working memory task than a low-SES girl about 64 times out of 100. This is a moderately large effect — though not as big as the difference in language-related scores, where the effect size of 0.95 means that we'd expect the middle-SES girl to have a higher score about 75% of the time.
From this information, you could make an informed guess as to whether I'm offering you a fair bet or not. (The answer is "not" — I'd expect to walk away with more than $800 of your money.)
This is not to say that Martha Farah's discovery about working memory was not meaningful and important. (It's probably less important than her discovery about linguistic abilities, but that's another matter.) Nevertheless, there's a serious point behind the foolishness about betting odds. Generic statements like "the working memories of children who have been raised in poverty have smaller capacities than those of middle-class children" seem to be general statements about the properties of groups, but in fact they're not. Rather, they're statements about the distribution of properties of the individuals making up the groups. And they're effectively meaningless if they don't tell us anything about how much the distributions overlap.
Whenever you see a statement of this kind, you should ask yourself "at what odds would I take a bet on the outcome of a series of random paired comparisons?" Or some related statement, like "how often would a randomly-selected member of group A beat a randomly-selected member of group B?", or "what is the effect size?" or "how do the quantiles compare?" In this particular case, the comparison was not only statistically significant, it involved a moderate effect size — but nothing about the Economist's description could give you any confidence in that outcome. It might have been a statistically-significant difference in group means with an effect size of 0.05, and I might have walked away with piles of your money at odds of 5-to-4 or less.
OK, let's go on with the article from The Economist:
Since Dr Farah’s discovery, Gary Evans and Michelle Schamberg of Cornell University have studied the phenomenon in more detail [and] have found that the reduced capacity of the memories of the poor is almost certainly the result of stress affecting the way that childish brains develop. […]
To measure the amount of stress an individual had suffered over the course of his life, the two researchers used an index known as allostatic load. This is a combination of the values of six variables: diastolic and systolic blood pressure; the concentrations of three stress-related hormones; and the body-mass index, a measure of obesity. For all six, a higher value indicates a more stressful life; and for all six, the values were higher, on average, in poor children than in those who were middle class. Moreover, because Dr Evans’s wider study had followed the participants from birth, the two researchers were able to estimate what proportion of each child’s life had been spent in poverty. That more precise figure, too, was correlated with the allostatic load.
The capacity of a 17-year-old’s working memory was also correlated with allostatic load. Those who had spent their whole lives in poverty could hold an average of 8.5 items in their memory at any time. Those brought up in a middle-class family could manage 9.4, and those whose economic and social experiences had been mixed were in the middle. [emphasis added]
Now it's betting time again. We're going to compare random pairs of poor and middle-class youths from population in this study, using the same working-memory test they used. I'm going to bet on the poor kids. At what odds will you take my bet?
This time, you should know enough to keep your wallet in your pants or purse until you get some more information. The cited paper (Gary W. Evans and Michell A. Schamberg, "Childhood poverty, chronic stress, and working memory", PNAS early edition, 3/30/2009) is Open Access, so you all can read it. Unfortunately, it doesn't really give us the information we need to set odds, at least not in an easy-to-assimilate form. But from this graph, you can get a sense of how it's going to turn out:
The difference in the group averages was 0.9 items. From the scatterplot, I'm guessing that the standard deviations were substantially greater than 1.8, so the effect size was less than the 0.5 that Farah et al. found; and as a result, I'm going to be happy to take your money at 2-to-1 odds. If I spent a bit of time toting up values from the graph and doing some simple calculations, I might be happy to lower the odds considerably and still be confident of coming out ahead.
More seriously, let's think about how the Economist's writer expressed this result. At least we're given some numbers, not just a generic statement that one group scores better than the other. But the only numbers that we get are the mean values for each group, without any way to judge what this means in terms of the degree of overlap of the distributions. And as a result, the numbers are almost meaningless, at least with respect to the issue that the article wants to address, namely "how poverty passes from generation to generation". (We might also want to know how well measures of working memory predict socio-economic outcomes; but never mind that for now.)
I'm not criticizing the research, which is an important contribution. (Martha Farah, in particular, is a good friend as well as a terrific researcher). Nor am I criticizing the Economist, exactly — it's not only the best magazine in the world these days, it's actually quite good. But on this particular point, the Economist is rarely any better than its peers.
This is presumably because a significant proportion of its readers would be baffled by talk of effect sizes or percentiles, while the proportion who are bothered by vague talk about generic differences is minuscule. Such things are not effectively taught or widely learned, even among quantitatively-minded intellectuals. But I also think that there's a linguistic aspect. If Benjamin Lee Whorf were alive, he might argue that our whole society is intellectually hamstrung by the way that English — like all the other languages of the world — tends to make us think about the evaluation and comparison of the properties of members of groups. And, I think, he might be right.
The easy and natural ways of talking about group comparisons express differences in terms of properties of the groups involved, or in terms of properties of imaginary generic or average group members: "the working memories of children who have been raised in poverty have smaller capacities than those of middle-class children"; "Those who had spent their whole lives in poverty could hold an average of 8.5 items in their memory at any time. Those brought up in a middle-class family could manage 9.4″. Writers and speakers may know what's really going on, at least with half of their brains, but readers and listeners are fooled into thinking that they understand these generic statements, even though in the absence of information about the comparison of distributions rather than the comparison of average values, they're left completely unable to put that understanding to any valid use.
This situation ought to be just as puzzling, at least to members of a more advanced civilization, as the Pirahã's ignorance of numbers is to us.