Replication of failure to replicate
« previous post |
Mouseover title: "Maybe encouraging the publication of null results isn't enough–maybe we need a journal devoted to publishing results the study authors find personally annoying."
Actually, there's a long history of scientific and scholarly publications based on personal annoyance — my favorite is the 1955-1961 back-and-forth between Herb Simon and Benoit Mandelbrot, discussed in "The long tail of religious studies?", 8/5/2010. And I have to confess that an occasional bit of annoyance has motivated some LLOG posts.
Anyhow, there's been some progress in relevant attitudes at journals, scientific and technical societies, and funders, towards promoting (and even requiring) the replication-friendly open publication of data, code, etc. — though there's still a long way to go…
A few relevant past posts:
"Open Data and Reproducible Research: Blurring the Boundaries between Research and Publication", Berlin 6 Open access Conference (11/12/2008)
"Human Language Technologies in the United States:Reflections 1966-2008", MYL Berlin 6 slides, 11/12/2008
"Reproducible research", 11/13/2008
"Reproducible Science at AAAS 2011", 2/18/2011
"Replication Rumble", 3/17/2012
"Textual narcissism", 7/13/2012
"Textual narcissism, replication 2", 7/14/2012
"Literate programming and reproducible research", 2/22/2014
“Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results”, NRC Workshop 2/26/2015
"Reliability", 2/28/2015
"Replicability vs. reproduciblity — or is it the other way around?", 10/31/2015
"Replicate vs. reproduce (or vice versa?)", 2/15/2018
Update — We should note that publishing open data and code is only one step towards a solution. In honest and intelligent research, there are still the problems of parameter choices, analysis method choices, and uncontrolled co-variates. And across the spectrum of motivated, biased, and less honest research, those problems get worse.
Still, access to data and code makes it easier to detect and fix such problems.