Language Log

« previous post | next post »

Nicholas Wade, "In Harvard Lab Inquiry, a Raid and a 3-Year Wait", NYT 8/13/2010, gives some additional information about the Marc Hauser scandal.  The new information is still basically rumor — the result of interviews with sources both anonymous and not, with the non-anonymous information being largely second hand. But it all suggests that whatever happened is more serious than just a bit of careless record-keeping.

Key points include a "raid" by "Harvard authorities::

Marc Hauser’s academic career was soaring when suddenly, three years ago, Harvard authorities raided his laboratory and confiscated computers and records.

There are apparently "eight charges", though it's not clear yet what they are:

In January this year, a faculty committee at last completed its report, said to contain eight charges against Dr. Hauser. But the report was kept secret and nothing changed until this month when someone showed The Boston Globe a letter about the investigation from Dr. Hauser to his faculty colleagues.

There's some general confirmation of the notion that monkeys can be very difficult subjects for the sort of experiments he does with them, and that the relevant data collection is often at best very subjective:

The captive animals, a colony of some 40 cotton-topped tamarins, may have contributed to the difficulties in Dr. Hauser’s laboratory. It is difficult to get the tamarins to pay attention, especially after the monkeys get used to experimenters.

“With some of these methods it was never clear to me how one could obtain meaningful results,” said a person with experience in Dr. Hauser’s lab, who requested anonymity for fear of retaliation. “The monkeys were often either jumping around, or not moving at all, and you rarely got the sense of an unambiguous response.”

A typical experimental paradigm is habituation/discrimination, where the monkeys are exposed to stimuli of a certain sort for a while, and then later exposed to a stimuli of a different sort. The dependent variable is some measure of how much attention they pay to the novel stimuli.  This measure is subjectively coded, by direct observation or by inspection of video recordings. To be believable, the coding should be "blind" — that is, the coders should not know which experimental category they are coding.  In the versions of such procedures used for studying auditory pattern perception in infants, precautions are generally taken to ensure that those who are coding a subject's attention are unaware of the nature of the trial being coded — see e.g. Janet Werker et al., "The conditioned head turn procedure as a method for testing infant speech perception", Early Dev Parent 6:171-8 (1997) . Unfortunately, this is by no means the norm in all areas of experimental psychology; in fact I'd say that "blind" coding of behavior is the exception rather than the rule, though I don't have counts to show this.

As far as I can recall, Hauser's papers don't specify blind coding. [Update — I was wrong, and should have checked first — e.g. Fitch and Hauser "Computational Constraints on Syntactic Processing in a Nonhuman Primate", Science 2004 says that the tamarins' "latency and duration of looking … were later scored blind to condition from the digitized video (>90% reliability)".]

If the  behavior to be coded is exceptionally hard to interpret, or even basically uninterpretable in some cases, then obviously this opens the door to a whole spectrum of problems, from the sort of "honest cheating" that tends to happen whenever subjective judgments are made by people who have beliefs about how the results should come out, to out-and-out fraud.

[If the behavior really was scored blind with >90% inter-annotator agreement, then there should be little or no reason for concern on this score. But then the quote in Wade's article (“With some of these methods it was never clear to me how one could obtain meaningful results. […] The monkeys were often either jumping around, or not moving at all, and you rarely got the sense of an unambiguous response.”) would be completely irrelevant. It's necessary to repeat again that we don't have very much evidence here, but the only concrete testimony that we do have — the 1995 mirror experiment and the withdrawn 2002 paper — says that at least the documentation of controlled scored of behavior was inadequate, and that in some cases the scoring could not be replicated from the video, or the video itself was missing.]

This is one of the reasons why (in my opinion) raw data should always be published.

Wade's article also gives confirmation of some (at best) careless record-keeping:

Other experimental problems have come to light with three articles investigated by the Harvard committee. In two, the supporting data did not exist. Dr. Hauser and a colleague repeated the experiments, and say they got the same results as published. In a third case, Dr. Hauser retracted an article published in the journal Cognition in 2002 but gave the editor no explanation of his reason for doing so.

And also confirmation (or at least presentation in print) of rumors that I earlier chose not to repeat, about the role of grad students and other junior lab members in bringing the problems to light:

Whatever the problems in Dr. Hauser’s lab, they eventually led to an insurrection among his staff, said Michael Tomasello, a psychologist who is co-director of the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany, and shares Dr. Hauser’s interest in cognition and language.

“Three years ago,” Dr. Tomasello said, “when Marc was in Australia, the university came in and seized his hard drives and videos because some students in his lab said, ‘Enough is enough.’ They said this was a pattern and they had specific evidence.”

My own feeling, for what it's worth, is that spectacular over-interpretation in the service of ambition ought to be as much a cause for concern as tendentious subjective data coding and poor record-keeping.  Putting it cynically, if you invent exciting explanations that extrapolate way beyond relatively pedestrian data, and you still get your papers published in Science and Nature, then maybe it doesn't matter so much whether your results are reproducible or even whether the data ever actually existed.

On the other side, Elyssa A. L. Spitzer, "Harvard Keeps Mum as Scientists Call for Transparency in Probe", Harvard Crimson 8/14/2010, quotes Steve Pinker in a very supportive vein:

“Marc is a beloved scientist, teacher, and colleague,” Hauser’s friend and colleague, psychology professor Steven A. Pinker wrote in an e-mail Thursday. Pinker has not collaborated with Hauser on any published studies. “He is widely admired not just for his astonishing breadth and creativity in devising ways to investigate deep problems with elegant experiments, but for his warmth, humor, and lack of pretension.”

Hauser did not respond to an interview request, but Pinker said his colleague will continue to head his lab next year while on leave.

August 14, 2010 @ 10:39 am · Filed by Mark Liberman under Psychology of language

Permalink