{"id":21956,"date":"2015-10-31T07:30:31","date_gmt":"2015-10-31T12:30:31","guid":{"rendered":"http:\/\/languagelog.ldc.upenn.edu\/nll\/?p=21956"},"modified":"2015-11-05T07:44:55","modified_gmt":"2015-11-05T12:44:55","slug":"replicability-vs-reproducibility-or-is-it-the-other-way-around","status":"publish","type":"post","link":"https:\/\/languagelog.ldc.upenn.edu\/nll\/?p=21956","title":{"rendered":"Replicability vs. reproducibility — or is it the other way around?"},"content":{"rendered":"

The term reproducible research<\/em><\/strong>, in its current sense<\/a>,\u00a0<\/em><\/strong>was\u00a0<\/em><\/strong>coined about 1990 by the geophysicist Jon Claerbout.\u00a0\u00a0<\/em><\/strong>Thus Jon Claerbout & Martin Karrenbach, \"Electronic Documents Give Reproducible Research<\/strong> a New Meaning<\/a>\", Society of Exploration Geophysics<\/em> 1992\u00a0[emphasis added, here and throughout]<\/em>:<\/p>\n

A revolution in education and technology transfer follows from the marriage of word processing and software command scripts. In this marriage an author attaches to every figure caption a pushbutton or a name tag usable to recalculate the figure from all its data, parameters, and programs. This provides a concrete definition of reproducibility<\/strong> in computationally oriented research. Experience at the Stanford Exploration Project shows that preparing such electronic documents is little effort beyond our customary report writing; mainly, we need to file everything in a systematic way. […]<\/span><\/p>\n

The principal goal of scientific publications is to teach new concepts, show the resulting implications of those concepts in an illustration, and provide enough detail to make the work reproducible. In real life, reproducibility<\/strong> is haphazard and variable. Because of this, we rarely see a seismology PhD thesis being redone at a later date by another person. In an electronic document, readers, students, and customers can readily verify results and adapt them to new circumstances without laboriously recreating the author's environment.<\/span><\/p>\n

<\/p>\n

I organized a session<\/a>\u00a0on \"Reproducible Research<\/strong>\"<\/a> at the Berlin 6 Open Access Conference<\/a>\u00a0in 2008; and Victoria Stodden organized a session entitled \"The Digitization of Science: Reproducibility<\/strong> and Interdisciplinary Knowledge Transfe<\/a>r\" at AAAS 2011 (LLOG coverage here<\/a>).<\/p>\n

Because research in Claerbout's lab mainly involved analysis of published seismological recordings collected and published by the USGS, the idea of re-doing an experiment by collecting new data\u00a0didn't ordinarily\u00a0arise — the closest thing would be what that paper calls \"adapting results to new circumstances\". And much the same situation obtains in other areas\u00a0where the goal is to model or explore large shared datasets, as is the case in most modern research in computational linguistics.<\/p>\n

But in many\u00a0other fields, it's natural to wonder whether an experiment would work if someone else tried to follow a similar recipe from start to finish. So at\u00a0some point between 1990 and 2006, people in this tradition\u00a0began using terms in the word family \u00a0replication replicable replicability<\/em><\/strong> to refer to the (traditional) process of completely re-running an experiment, with\u00a0all the effects of new researchers, new equipment, new subjects or other raw materials, etc.\u00a0Thus Roger Peng et al., \"Reproducible<\/strong> Epidemiologic Research<\/a>\", American Journal of Epidemiology<\/em> 2006:<\/p>\n

The replication<\/strong> of important findings by multiple independent investigators is fundamental to the accumulation of scientific evidence. Researchers in the biologic and physical sciences expect results to be replicated<\/strong> by independent data, analytical methods, laboratories, and instruments. Epidemiologic studies are commonly used to quantify small health effects of important, but subtle, risk factors, and replication is of critical importance where results can inform substantial policy decisions. However, because of the time, expense, and opportunism of many current epidemiologic studies, it is often impossible to fully replicate<\/strong> their findings. An attainable minimum standard is \u201creproducibility<\/strong>,\u201d which calls for data sets and software to be made available for verifying published findings and conducting alternative analyses. The authors outline a standard for reproducibility<\/strong> and evaluate the reproducibility<\/strong> of current epidemiologic research. They also propose methods for reproducible research<\/strong> and implement them by use of a case study in air pollution and health.<\/span><\/p>\n

For another example of the same terminological tradition, see \"Replication, psychology, and Big Science<\/a>\", Simply Statistics<\/em> 2012:<\/p>\n

A study is reproducible<\/strong> if there is a specific set of computational functions\/analyses (usually specified in terms of code) that exactly reproduce all of the numbers in a published paper from raw data.\u00a0It is now recognized that a critical component of the scientific process is that data analyses can be reproduced. This point has been driven home particularly for personalized medicine applications, where irreproducible results\u00a0can lead to delays<\/a>\u00a0in evaluating new procedures that affect patients\u2019 health.\u00a0<\/span><\/p>\n

But just because a study is reproducible<\/strong> does not mean that it is\u00a0replicable<\/em><\/strong>. Replicability<\/strong> is stronger than reproducibility<\/strong>. A study is only replicable<\/strong> if you perform the exact same experiment (at least) twice, collect data in the same way both times, perform the same data analysis, and arrive at the same conclusions. The difference with reproducibility<\/strong> is that to achieve replicability<\/strong>, you have to perform the experiment and collect the data again. This of course introduces all sorts of new potential sources of error in your experiment (new scientists, new materials, new lab, new thinking, different settings on the machines, etc.)<\/span><\/p>\n

And there's a substantial and growing literature on (computational and social) methods for achieving reproducibility<\/strong> in Claerbout's sense, and replicability<\/strong> in Peng's sense.\u00a0An important recent survey of the movement\u00a0is Victoria Stodden et al., Eds., Reproducible Research<\/a><\/em><\/strong>, Taylor & Francis 2014:<\/p>\n

Science moves forward when discoveries are replicated<\/strong> and reproduced<\/strong>. In general, the more frequently a given relationship is observed by independent scientists, the more trust we have that such a relationship truly exists in nature. Replication<\/strong>, the practice of independently implementing scientific experiments to validate specific findings, is the cornerstone of discovering scientific truth. Related to replication<\/strong> is reproducibility<\/strong>, which is the calculation of quantitative scientific results by independent scientist using the original datasets and methods. Reproducibility<\/strong> can be thought of as a different standard of validity because it forgoes independent data collection and uses the methods and data collected by the original investigator. Reproducibility<\/strong> has become an important issue for more recent research due to advances in technology and the rapid spread of computational methods across the research landscape.<\/span><\/p>\n

Clear enough, right?<\/p>\n

But more recently, some researchers have started using\u00a0the same terms with the reference more or less switched. I think that this confusion originates with\u00a0Chris Drummond, \"Replicability<\/strong> is not Reproducibility<\/strong>: Nor is it Good Science<\/a>\", ICML 2009:<\/p>\n

At various machine learning conferences, at various times, there have been discussions arising from the inability to replicate<\/strong> the experimental results published in a paper. There seems to be a wide spread view that we need to do something to address this problem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility<\/strong> of experimental results is the hallmark of science. Therefore, given that most of us regard machine learning as a scientific discipline, being able to replicate<\/strong> experiments is paramount. I want to challenge this view by separating the notion of reproducibility<\/strong>, a generally desirable property, from replicability<\/strong>, its poor cousin. I claim there are important differences between the two. Reproducibility<\/strong> requires changes; replicability<\/strong> avoids them. Although reproducibility<\/strong> is desirable, I contend that the impoverished version, replicability<\/strong>, is one not worth having.<\/span><\/p>\n

And Drummond's confusion has been picked up a few others — e.g. Arturo Casadevall & Ferric Fang, \"Reproducible<\/strong> Science<\/a>\", Infection and Immunity<\/em> 2010:<\/p>\n

Although many biological scientists intuitively believe that the reproducibility<\/strong> of an experiment means that it can be replicated<\/strong>, Drummond makes a distinction between these two terms. Drummond argues that reproducibility<\/strong> requires changes, whereas replicability<\/strong> avoids them. In other words, reproducibility<\/strong> refers to a phenomenon that can be predicted to recur even when experimental conditions may vary to some degree. On the other hand, replicability<\/strong> describes the ability to obtain an identical result when an experiment is performed under precisely identical conditions.<\/span><\/p>\n

Or Thilo Mende, \"Replication<\/strong> of Defect Prediction Studies: Problems, Pitfalls and Recommendations<\/a>\", PROMISE 2010:<\/p>\n

In the early days, […]\u00a0most prediction models were based on proprietary data, thus preventing independent replication<\/strong>. With the rise of the PROMISE repository1 , this situation has changed. This repository collects publicly available data sets, the majority of them for the task of defect prediction. Currently, there are more than 100 such data sets inside the PROMISE repository, and many more are made available elsewhere. <\/span><\/p>\n

This trend is very beneficial, as it enables researchers to independently verify or refute previous results. Drummond argues that replication<\/strong> \u2014 the repetition of an experiment without any changes \u2014 is not worthwhile. He favors reproducing<\/strong> experiments with changes, since only this adds new insights. While we agree that the pure replication<\/strong> of experiments on the same data sets should not lead to new results, we argue that replicability<\/strong> is nevertheless important: When applying previously published procedures to new data sets, or new procedures to well-known data sets, researchers should be able to validate their implementations using the originally published results.<\/span><\/p>\n

This confusion seems to have led some researchers to reject the whole distinction — e.g. Brian Nosek, \"An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility<\/strong> of Psychological Science<\/a>\", Perspectives on Psychological Science<\/em> 2012:<\/p>\n

Some distinguish between \u201creproducibility<\/strong>\u201d and \u201creplicability<\/strong>\u201d by treating the former as a narrower case of the latter (e.g., computational sciences) or vice versa (e.g., biological sciences). We ignore the distinction.<\/span><\/p>\n

(As the citations in this post suggest, and as a little poking around in Google Scholar will confirm, Nosek's notion that\u00a0this is a difference between Computer Science and Biology is false. As far as I can tell, it's a difference between people influence by Drummond's provocative but deeply confused article, and everybody else in a dozen different fields — though maybe there has been some independent invention of related confusions as well.)<\/p>\n

It seems to me that<\/p>\n

    \n
  1. Under whatever names, the distinction between replicability<\/strong> and reproducibility<\/strong> is worth preserving (and indeed extending — see below);<\/li>\n
  2. Since the technical term \"reproducible research<\/strong>\" has been in use since 1990, and the technical distinction between reproducible<\/strong> and replicable<\/strong> at least since 2006, we should reject Drummond's 2009 attempt to\u00a0re-coin\u00a0technical terms reproducible<\/strong> and replicable<\/strong>\u00a0in senses that assign the terms to concepts nearly opposite to those used in the definitions\u00a0by Claerbout, Peng and others.<\/li>\n<\/ol>\n

    Why preserve\u00a0the distinction in an extended or elaborated form? Because there are many variations on the theme, all of them sometimes worthwhile. We might re-apply the original\u00a0computational analysis\u00a0to the original data, perhaps to\u00a0check unreported aspects of the method or the results; we might re-implement the model or algorithm and apply it to the original data, to test a new program or to check for coding or algorithmic errors; we might apply the original computational analysis to new data, meant to test exactly the same hypotheses; we might apply a different model or algorithm to the original data, as an independent\u00a0test of the original hypothesis, or in support of a different one; we might apply the original computational analysis to new data focused on an analogous but systematically different set of questions; and so on. Perhaps the most common and most valuable variation is benchmarking alternative models or algorithms with respect to the same quantitative evaluative metric on the same training and testing material. All of these varied responses to a publication are consistent with Jon Claerbout's original vision, in my opinion, though they go far beyond\u00a0\u00a0simply \"[attaching] to every figure caption a pushbutton or a name tag usable to recalculate the figure from all its data, parameters, and programs\".<\/p>\n

    For many\u00a0reasons, I think that Drummond is profoundly wrong on the substance. But even if you believe his assertion that\u00a0\"Although X\u00a0is desirable, I contend that the impoverished version, Y, is one not worth having\", you should reject his attempt to swap the reference\u00a0of the terms X and Y, substituting reproducibility<\/strong><\/em> for what many others have been calling replicability<\/strong><\/em>, and vice versa.<\/p>\n

    Some previous LLOG posts on related topics:\u00a0\u00a0\"Reproducible research<\/a>\" (11\/14\/2008); \"Reproducible Science at AAAS 201<\/a>1\" (2\/18\/2011); \"Literate programming and reproducible research<\/a>\" (2\/22\/2014); and \"Reliability<\/a>\" (2\/28\/2015).<\/p>\n","protected":false},"excerpt":{"rendered":"

    The term reproducible research, in its current sense,\u00a0was\u00a0coined about 1990 by the geophysicist Jon Claerbout.\u00a0\u00a0Thus Jon Claerbout & Martin Karrenbach, \"Electronic Documents Give Reproducible Research a New Meaning\", Society of Exploration Geophysics 1992\u00a0[emphasis added, here and throughout]: A revolution in education and technology transfer follows from the marriage of word processing and software command scripts. […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[8],"tags":[],"class_list":["post-21956","post","type-post","status-publish","format-standard","hentry","category-the-language-of-science"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/posts\/21956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=21956"}],"version-history":[{"count":23,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/posts\/21956\/revisions"}],"predecessor-version":[{"id":22052,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=\/wp\/v2\/posts\/21956\/revisions\/22052"}],"wp:attachment":[{"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=21956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=21956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/languagelog.ldc.upenn.edu\/nll\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=21956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}