Why most (science) news is false
« previous post | next post »
François Gonon et al., "Why Most Biomedical Findings Echoed by Newspapers Turn Out to be False: The Case of Attention Deficit Hyperactivity Disorder", PLoS ONE 9/12/2012:
Methods: We focused on attention deficit hyperactivity disorder (ADHD). Using Factiva and PubMed databases, we identified 47 scientific publications on ADHD published in the 1990s and soon echoed by 347 newspapers articles. We selected the ten most echoed publications and collected all their relevant subsequent studies until 2011. We checked whether findings reported in each “top 10” publication were consistent with previous and subsequent observations. We also compared the newspaper coverage of the “top 10” publications to that of their related scientific studies.
Results: Seven of the “top 10” publications were initial studies and the conclusions in six of them were either refuted or strongly attenuated subsequently. The seventh was not confirmed or refuted, but its main conclusion appears unlikely. Among the three “top 10” that were not initial studies, two were confirmed subsequently and the third was attenuated. The newspaper coverage of the “top 10” publications (223 articles) was much larger than that of the 67 related studies (57 articles). Moreover, only one of the latter newspaper articles reported that the corresponding “top 10” finding had been attenuated. The average impact factor of the scientific journals publishing studies echoed by newspapers (17.1 n = 56) was higher (p<0.0001) than that corresponding to related publications that were not echoed (6.4 n = 56).
This will not be a surprise to any honest working scientists, nor to members of the public who have observed the fate of science and technology in the media ecosystem over the years.
Gonon et al. focus on the role of publication bias and sensationalism at top scientific journals — but of course popular media have their own motivations, which often lead to credulous trumpeting of "results" that were never published in the technical or scientific literature at all, much less featured in a high-impact-factor journal. For a random sample of past LL posts on various aspects of these phenomena, see "Quit email, get smarter?" (4/23/2005), "The Agatha Christie Code" (12/26/2005), "It's always Silly Season in the BBC science section" (8/26/2006), "Flacks and hacks and brainscans" (11/23/2007), "David Brooks, Social Psychologist" (8/13/2008), "Debasing the coinage of rational inquiry" (4/22/2009), "The business of newspapers is news" (12/10/2009), "Texting and language skills" (8/2/2012), etc.
[Update — for some discussion of the within-science aspects of this issue, see Howard Wainer, "The survival of the fittists", American Scientist, September/October 2012.]
Mr Fnortner said,
September 21, 2012 @ 7:48 am
Coupled with the likely bias toward false findings in published research results generally (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124), this study puts the nail in the coffin: Don't look to the popular press for scientific information–everything is wrong.
Jonathon said,
September 21, 2012 @ 10:36 am
I little while ago I saw a forum discussion about the fact that science news is always riddled with inaccuracies. Someone suggested doing a study on science news reporting and wondered how that study would be reported. Someone else suggested, "New study proves journalism is pretty accurate when it comes to science reporting."
L said,
September 21, 2012 @ 11:29 am
Unfair example. It likely attracted reporters and editors who had a special interest in ADHD and were unable to focus suffiently in order to oh look a shiny thing.
Eric P Smith said,
September 21, 2012 @ 3:25 pm
For many years I have despaired at the standard of science reporting in the newspapers, and I have collected some examples. My candidate for the title of least accurate headline in the history of British broadsheet publishing is the Guardian headline of 10 October 2002. It began:
For monkey read ape. For man read hominid (or hominin in more a recent classification system). For our oldest ancestor read the oldest known hominid. Depending on how you count them, three if not four serious errors in 9 words.
Eric P Smith said,
September 21, 2012 @ 6:32 pm
…in a more recent…
Brett said,
September 21, 2012 @ 7:53 pm
@Eric P Smith: Writing "our oldest ancestor" is pretty egregious. I once mentioned at the dinner table that the oldest thing that could be identified as one of our ancestors, in the sense one would normally think of, would be a sort of sponge. My daughter sarcastically pointed out that she did not want to attend any reunions with that branch of our family.
However, I don't agree with your characterization of "Monkey or man?" From a scientific point of view, every hominid is an ape, and every ape is a monkey. So the question is not well posed, but not for the reasons you seem to suggest.
Chad Nilep said,
September 21, 2012 @ 8:48 pm
@Mr Fnorter
According to Gonon et al., two of the ten stories reported in newspapers turned out to be correct. So it's not quite true that "everything is wrong".
Though now that I think about it, I guess that sort of proves your point. Never mind; carry on.
the other Mark P said,
September 22, 2012 @ 6:03 am
This will not be a surprise to any honest working scientists,
You think? I am not so sure.
In psychology, medicine, economics, etc then probably you are right. Most workers in the field will doubt most of the published works.
In fields where you are not allowed to be "incorrect", I think you will find that scientists will react very badly to being told that most of the published findings are wrong.
Fields where you are not allowed to be incorrect include (IMO obviously), education, genetic inheritance when it applies to intelligence or similar by humans and climate science. In such fields I have seen that any suggestion that the approved view is wrong is punished extremely severely by the majority of scientists.
(Cue outrage that I suggest that climate scientists are mostly wrong. They can't help it — no matter how much evidence that most science is wrong, there are some that cannot be wrong.)
[(myl) The assertion in this post was that most science news is wrong. The claimed reason is that scientists and (especially) journals have a "publication bias" in favor of positive as opposed to negative results, and also (especially high-impact journals) in favor of unexpected or spectacular results; while journalists and their editors have an even stronger bias in favor of striking results, combined with poor understanding of scientific writing and a willingness to reprint press releases.
All of that means that science news is biased towards erroneous and over-interpreted results, which (by regression to the mean and the general adversarial process of scientific discourse) are especially likely to turn out fairly quickly to be wrong. This is completely different from saying that most published science is wrong. There are some factors leading in that direction as well — especially publication bias — but these factors are much weaker overall than they are in the case of mass-media science news.
And it's true, by design, that even the best science often turns out to be "wrong", in the sense of being an approximation that is eventually replaced by a more complete or more insightful view. But this happens for different reasons and on a much longer time scale.]
Andy Averill said,
September 22, 2012 @ 8:27 am
I think by now we all have pretty low expectations of the quality of science news in non-specialist publications. But really, why should it be so bad? I would assume that a job as a science reporter in a major media outlet would be highly sought after. With all the unemployed science PhD's out there these days, it ought to be easy to find someone who really knows what they're doing.
And yet time after time we see a total misunderstanding of the content of a particular piece of research, as well as a generally poor conception of how science actually works. Is it that the editors of the publications are so ignorant themselves that they can't tell the difference between real science and junk? Or is the competition for space on the front page so intense that they just cynically throw the most eye-catching stuff out there and hope nobody can tell the difference? In which case it's easy to see why they don't bother with a follow-up when something gets refuted later on.
L said,
September 22, 2012 @ 12:17 pm
What's equally scary is the possibility that science is not unique in this.
How many journalists do a whole better on politics, say, or anything else that makes headlines? Indeed, headlines of themselves are pretty damn suspect.
Eric P Smith said,
September 22, 2012 @ 1:12 pm
@Brett
That depends on what you mean by an ape. It would be nice if every taxonomic term described a monophyletic group (a species together with all its descendants), but that does not always happen in practice. Some biologists use the term "ape" to exclude humans, others to include humans.
The word "monkey", as far as I know, is never used by biologists to include apes. Monkeys do not form a monophyletic group.
joanne salton said,
September 22, 2012 @ 2:05 pm
Well, look on the bright side. We used to almost totally rely on people who published only for financial gain, and now we don't have to.
the other Mark P said,
September 22, 2012 @ 3:15 pm
The assertion in this post was that most science news is wrong.
The conclusion reached can be that the media is largely to blame. And it's largely true that hey don't take time to check against other sources, or for prior liklihood.
I, however, blame the scientists – or at the very least their institutions. Incorrect stories are not corrected properly. The very act of sending out a press release on a scientific story suggests that publicity is the main aim, not truth.
And that's in objective disciplines. In politicised fields the main aim is to persuade anyway, not reach any objective standard of truth. In those fields almost all the "research" is junk.
Andy Averill said,
September 23, 2012 @ 11:43 am
@L, you're right, there's plenty of bad political reporting out there. But there are enough checks and balances that eventually a reasonably assiduous reader can figure out what's really going on. With science, it's a lot harder to do that if you only have access to general-purpose publications.
Kenny Easwaran said,
September 24, 2012 @ 12:09 pm
I'm not sure that this is the relevant comparison to make. It's natural that if you take the 10 most reported on studies, and compare them to a group of 67 other studies, then regardless of whether the set are connected, the 10 most repeated ones will be reported more than the 67 others. What would be more useful would be to look at all initial studies, and compare them to all follow-ups. Or at least do some sort of test where you don't build in the fact that one group is the most reported on. Even just take a random sample of news reporting, and see how much is initial studies that are eventually discredited, and how much is devoted to later studies that moderate the findings of initial studies – that would be biased towards reporting equally on both sides.
[(myl) It's possible that the least-reported studies in the group were even more likely to be disproven or amended than the most-reported studies — but the point of this exercise was to show that the most-reported stories are not very reliable.
There's some evidence that journal impact factor is positively correlated with bias (and thus negatively correlated with replicability) — see M R Munafò et al. "Bias in genetic association studies and impact factor", Molecular Psychiatry 14: 119–120, 2009; discussed in "The business of newspapers is news", 12/10/2009.]