I'm at the AAAS 2011 meeting in DC, mainly because as chair-elect of Section Z (Linguistics) I'm duty-bound to be here, but also partly because I'm giving a talk in a symposium tomorrow afternoon on "The Digitization of Science: Reproducibility and Interdisciplinary Knowledge Transfer". The session was organized by Victoria Stodden, and this is its abstract:
Scientific computation is emerging as absolutely central to the scientific method, but the prevalence of very relaxed practices is leading to a credibility crisis affecting many scientific fields. It is impossible to verify most of the results that computational scientists present at conferences and in papers today. Reproducible computational research, in which all details of computations — code and data — are made conveniently available to others, is a necessary response to this crisis. This session addresses reproducible research from three critical vantage points: the consequences of reliance on unverified code and results as a basis for clinical drug trials; groundbreaking new software tools for facilitating reproducible research and pioneered in a bioinformatics setting; and new survey results elucidating barriers scientists face in the practice of open science as well as proposed policy solutions designed to encourage open data and code sharing. A rapid transition is now under way — visible particularly over the past two decades — that will finish with computation as absolutely central to scientific enterprise, cutting across disciplinary boundaries and international borders and offering a new opportunity to share knowledge widely.
Victoria Stodden's blog, though not updated frequently, is worth reading. One post that I especially enjoyed was a discussion of HackNY, a summer program aiming to "'get the kids off the street' by giving them alternatives to entering the finance profession".
My own contribution to today's symposium is "Lessons for Reproducible Science from the DARPA Speech and Language Program":
Since 1987, DARPA has organized most of its speech and language research in terms of formal, quantitative evaluation of computational solutions to well-defined "common task" problems. What began as an attempt to ensure against fraud turned out to be an extraordinarily effective way to foster technical communication and to explore a complex space of problems and solutions. This engineering experience offers some useful (if partial) models for reproducible science, especially in the area of data publication; and it also suggests that the most important effects may be in lowering barriers to entry and in increasing the speed of scientific communication.
Update — Victoria Stodden has put the slides and other links from the presentations up on the web here.
The first talk in the session offered an especially vivid explanation of why "reproducible research" is a consequential slogan: Keith Baggerly, "The Importance of Reproducible Science in High-Throughput Biology: Case Studies". His abstract:
High-throughput biological assays let us ask very detailed questions about how diseases operate, and promise to let us personalize therapy. Data processing, however, is often not described well enough to allow for reproduction, leading to exercises in “forensic bioinformatics” where raw data and reported results are used to infer what the methods must have been. Unfortunately, poor documentation can shift from an inconvenience to an active danger when it obscures not just methods but errors.
In this talk, we examine several related papers using array-based signatures of drug sensitivity derived from cell lines to predict patient response. Patients in clinical trials were allocated to treatment arms based on these results. However, we show in several case studies that the reported results incorporate several simple errors that could put patients at risk. One theme that emerges is that the most common errors are simple (e.g., row or column offsets); conversely, it is our experience that the most simple errors are common. We briefly discuss steps we are taking to avoid such errors in our own investigations.
The presentation was chilling. For an earlier version of the same talk — with a video of the lecture synchronized to his slides — see here. (This is also a good example of what, in my opinion, the AAAS should do with the presentations in the >150 symposiums in each annual meeting, in place of the high-1980s technology of selling audio CDs.)