There's an article in the current issue of The Economist that you should read carefully: "Trouble at the lab", 10/19/2013. If you're a regular reader of Language Log, you'll be familiar with the issues that it raises — lack of replication, inappropriate use of "statistical significance" testing, data dredging, the "file drawer effect", inadequate documentation of experimental methods and data-analysis procedures, failure to publish raw data, the role of ambition and ideology, and so on.
But all the same, I'm going to push back. The problems in science, though serious, are nothing new. And the alternative approaches to understanding and changing the world, including journalism, are much worse. In fact, some of the worst problems in science are the direct result, in my opinion, of the poor quality of science journalism. One of the key reasons that leading scientific journals publish bad papers is that both the authors and the editors are looking for media buzz, and can usually count on the media to oblige.
The article leads with a quote from Daniel Kahneman's open letter on problems with replication in social priming research –"I see a train wreck looming" — and generalizes his apprehension to the scientific enterprise as a whole, especially to the biomedical area. There's a beautifully clear explanation of John Ioannidis's statistical argument that a large proportion of published research findings are likely to be false, summarized in this graphic (or see this animation):
The article goes on to discuss inadequate "blinding" and the problem of confirmation bias in dataset creation, and the many opportunities that "Big Data" offers for researchers to fool themselves as well as others. There's an excellent discussion of the nature and status of replication in various sciences. And the article omits a few criticisms that cut even deeper, such as the demonstration that replicability is inversely correlated with impact factor.
Hanging over everything else in the article, there's an implicit threat of financial retaliation:
The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America’s National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.
But the private-sector part of the biomedical research enterprise suffers from all the same problems as the "public part of the process", perhaps to an even greater extent. And surely the appropriate way to evaluate public funding of research, biomedical and otherwise, is to look at its overall impact on a time scale of decades, not at the quality of individual research reports.
In the biomedical area specifically, $59 billion is only about 0.1% of the OECD's total 2012 GDP, whereas health care costs are running between 7% and 17% of OECD countires' GDP. The right question to ask about this research investment is not what proportion of published results can be replicated, but how much solid understanding and effective intervention emerges from the overall process. I'll leave it to others to make this judgment about biomedical research — but I'm confident that the much smaller public investment in areas that I know more about has been repaid many times over.
On the other hand, I agree that the poor quality of much published research is a problem worth trying to fix. My own opinion is that trying to tighten up on pre-publication review will make the problems worse, not better, and that the key to improvement is to focus on the things that happen after publication.
These include (informed) discussion, open examination of (obligatorily published) data and analysis code, attention paid to (non) replication efforts, and so on. All of this happens to some extent now — but it's comparatively starved of resources and attention. Most published studies now don't publish their experimental materials or or their raw data or the computer scripts that generate their cited numbers and tables and graphs, even when there's no serious impediment to doing so. Many important journals will not publish failures to replicate, apparently as a matter of principle, and give little or no serious attention to serious and expert objections to problematic published papers.
And researchers need to be reminded, early and often, of Feynman's observation that "The first principle is that you must not fool yourself", and of Hamming's warning that you should "beware of finding what you're looking for".
For those readers who would like to prolong their mood of righteous indignation, here's a small sample of previous LL posts on related topics:
"The secret sins of academics", 9/16/2004
"The apotheosis of bad science at the BBC", 5/27/2007
"Listening to Prozac, hearing effect sizes", 3/1/2008
"Winner's curse", 10/15/2008
"The business of newspapers is news", 9/10/2009
"The Happiness Gap is back is back is back is back", 9/20/2009
"We Need More Bad Science Writers", 6/7/2010
"'Vampirical' hypotheses", 4/28/2011
"Why most science news is false", 9/21/2012
"The open access hoax and other failures of peer review", 10/5/2013
"Annals of overgeneralization", 10/8/2013
Or see Geoff Pullum, "The Bad Science Reporting Effect", The Chronicle of Higher Education 3/15/2012.