Suppose you heard about a study "showing" that Ivy League students are more socially sensitive than students at public universities or students at private colleges not among the Ancient Eight. You'd be skeptical, I hope.
So you take a look at the study, and discover that the authors — themselves Ivy League grads — did five experiments.
In the first experiment, they chose three Harvard students who exemplify, in their opinion, the best characteristics of that fine institution, and three students from the University of Michigan, again selected to represent the authors' idea of what such students should be like. They then subjected these six students to a battery of tests of empathy and social intelligence, and found that the three Harvard students scored a bit better than the three Michigan students.
The other four experiments were similar. In the second experiment, the authors selected three Princeton students from among a few dozen student-government leaders, and compared them to three selected representatives of the University of Oregon football team, and three (in their opinion characteristic) young people who did not attend college at all. Experiment 3 tested six new students, three from Yale and three from the University of Arizona, again selected to represent the authors' opinion of what such students should be like. Experiment 4 re-used four of the students from Experiment 3, but substituted two new choices from the same pools. And Experiment 5 re-used five of the six students from Experiment 4, substituting for one participant who seemed on reflection not to be quite of the Right Kind.
At this point, you should be saying to yourself, Wait a minute, this is a total crock! Where was it published, in one of those fake take-the-money-and-run open-access journals?
Read the rest of this entry »