Language Log

« previous post | next post »

Andrew Gelman, "Freakonomics does it again (not in a good way). Jeez, these guys are credulous", Statistical Modeling, Causal Inference, and Social Science 10/28/2024:

From the team that brought you “good-looking parents are 36% more likely to have a baby daughter as their first child than a baby son” and “The PDO cool mode has replaced the warm mode in the Pacific Ocean, virtually assuring us of about 30 years of global cooling” (background here and here) comes a new nugget of 24-carat credulity:

I [economist and author Steven Levitt] cannot think of an academic whose research findings have more consistently surprised me than my guest today, Harvard psychologist Ellen Langer. She’s a scientist, but her results seriously challenge the beliefs of mainstream science.

If the findings consistently surprise you, and they seriously challenge the beliefs of mainstream science, then maybe you should more seriously consider the possibility that these findings are wrong! Langer’s much-publicized work has been questioned before (for example, here and here).

Andrew quotes Levitt further:

I’ve read the work of many scholars and I can honestly say that you win the prize for the body of research that most consistently finds results that are completely the opposite of what I would have predicted. You and I have completely different models of how the world works. And the data keep supporting your model. . . .

… and then responds:

That’s not true! The data don’t keep supporting Langer’s model. Just for starters, see the two links given above, or this discussion by linguist Mark Liberman from 2009). Or my recent paper with Nick Brown. Or this recent post at The Skeptic.

This is a big, big problem. If you come into the discussion with the question, “How do you know whether you should believe surprising results?”, then you can’t just say, “the data keep supporting your model.” By making this unsupported claim about the data, you’ve given up your investigation before you’ve even begun!

The details in this case are complicated, not to say mind-numbing, so I won't be surprised if you give the whole thing a pass, including my long 2009 discussion. But Andrew's conclusion is worth reading and thinking about:

I write some of this in a jocular tone, only because that’s one way for me to deal with it. I laugh because that’s better than crying. In all seriousness, I think that experimental science can improve our lives, and it frustrates me when bad science takes up the space that could be occupied by good science.

And Steven Levitt . . . he’s got the training and experience to evaluate scientific claims! He could read Langer’s papers, he could download and reanalyze what data are available, he could google search for replications and criticisms, he could read what Mark Liberman and others had to say, etc. He doesn’t have to do that work, but, if he’s gonna ask, “How do you know whether to believe surprising results?”, then he should. Otherwise, why bother? Langer’s been interviewed a million times already; what’s the point of one more puff piece? I just don’t get it. Levitt can play a useful role in the conversation here, and he chooses not to. Really frustrated.

Again, I think it’s just fine that Langer and her colleagues do their research—not that they need my approval or endorsement! Speculative studies are part of the research ecosystem. My problem is not with high-risk, potential high-return research; my problem is the misrepresentation of scientific evidence.

My hypothesis about why "Levitt can play a useful role in the conversation here, and he chooses not to" is a simple one. It's for the same reason that the BBC misrepresented and even fabricated evidence about Himba color perception, in their spectacularly fraudulent 2011 documentary "Do you see what I see?". See "It's not easy seeing green" (3/2/2015) and "Himba color perception" (3/17/2015).

Levitt isn't in the business of evaluating scientific experiments, he's in the business of mass-audience journalism. He wants clicks, and people will be interested, for good reason, in a Freakonomics episode that promises to tell them about how we can "improve our physical health by changing our mind".

October 31, 2024 @ 8:19 am · Filed by Mark Liberman under Language and the media, The language of science

Permalink