Language Log

Lorena Barba, "Terminologies for Reproducible Research", arXiv.org 2/9/2018:

Reproducible research—by its many names—has come to be regarded as a key concern across disciplines and stakeholder groups. Funding agencies and journals, professional societies and even mass media are paying attention, often focusing on the so-called "crisis" of reproducibility. One big problem keeps coming up among those seeking to tackle the issue: different groups are using terminologies in utter contradiction with each other. Looking at a broad sample of publications in different fields, we can classify their terminology via decision tree: they either, A—make no distinction between the words reproduce and replicate, or B—use them distinctly. If B, then they are commonly divided in two camps. In a spectrum of concerns that starts at a minimum standard of "same data+same methods=same results," to "new data and/or new methods in an independent study=same findings," group 1 calls the minimum standard reproduce, while group 2 calls it replicate. This direct swap of the two terms aggravates an already weighty issue. By attempting to inventory the terminologies across disciplines, I hope that some patterns will emerge to help us resolve the contradictions.

Following up on "Citation crimes and misdemeanors" (9/9/2017), Breffni O'Rourke sent in a link to Michel Paradis, "More belles infidèles — or why do so many bilingual studies speak with forked tongue?", Journal of Neurolinguistics 2006:

This note reports misquotations, misinterpretations, misrepresentations, inaccuracies and plain falsehoods found in the literature on the neuroscience of bilingualism. They are astounding in both number and kind. Authors cite papers that do not exist, or that exist but are absolutely irrelevant to, or even occasionally argue against, the point they are cited to support; or they attribute a statement to the wrong source, sometimes to a person who has vehemently and persistently argued against it. Obvious errors are quoted for years by numerous authors who have not read the original paper, until somebody blows the whistle — and even then, some persevere. As Darwin [Darwin, C. (1872). The origin of species. 6th edition. New York: A. L. Burt.] put it: ‘great is the power of steady misrepresentation’.

Terry Provost wrote to express interest in the topic of "citation plagiarism", linking to a couple of Bill Poser's LLOG posts ("Citation plagiarism", 6/15/2007; "Citation Plagiarism Once Again", 4/23/2008), and noting that "yours was one of very few mentions of the topic I found". Provost points to a somewhat more recent article on a related topic (Charlie Tyson, "Academic Urban Legends", Inside Higher Ed 8/6/2014), and added "Bottom line, I think the subject is quite important, as concerns things like the Jick letter, NEJM".

That's a reference to a letter reporting only four cases of addiction in 11,882 hospital patients who were given narcotics: Jane Porter and Hershel Jick, "Addiction rare in patients treated with narcotics", New England Journal of Medicine 1980. The idea seems to be that a harmful conclusion was spread by people who cited the letter without considering its content — see Taylor Haney, "Doctor Who Wrote 1980 Letter On Painkillers Regrets That It Fed The Opioid Crisis", NPR 6/16/2017.

I'm following up on this note because Bill Poser's old LLOG post no longer accepts comments, and so Terry Provost added his remarks as a comment on a randomly selected recent article, which is something we discourage. This new post gives Mr. Provost a chance to say his piece. (The reason for closing comments on old articles is that we were logging about 10,000 spam comments per day, before we closed comments on posts more than a couple of weeks old. We still get plenty of spam comments, but the number is more manageable, since there are fewer targets. )

More than 2,400 researchers have now signed an "Open Letter to NIH Director Francis Collins" that starts like this:

We are writing to request that NIH delay implementation of its policy that sweeps basic science into a clinical trials framework until adequate feedback about its impact is obtained from the affected scientific community. We wholeheartedly agree with NIH’s goals of increasing scientific transparency and rigor, but we ask that you consider alternative mechanisms to accomplish those goals that would have fewer adverse effects on basic research.

The background is a new definition of what counts as a "clinical trial", to be enforced starting 1/1/2018 ("NIH's Definition of a Clinical Trial"):

A research study in which one or more human subjects are prospectively assigned to one or more interventions (which may include placebo or other control) to evaluate the effects of those interventions on health-related biomedical or behavioral outcomes.

Interpreted literally, this means that a study of priming effects on speech perception in healthy undergraduate students might count as a "clinical trial", since "human subjects are prospectively assigned to one or more interventions" (the priming part), and speech perception is a "health-related biomedical or behavioral outcome". Or maybe not. NIH has given some bizarrely irregular examples of how to interpret this rather general definition — thus vision and memory in adults are apparently "health-related outcomes" but learning in children is not.

The Daily Telegraph did not do much for its reputation, at least in my eyes, when it confused the defense with the prosecution after a celebrity sexual assault mistrial. Nor when it recently consulted me about whether there were grammar mistakes on a banknote, learned that there clearly were not, but went ahead and published the claim that there were anyway. Now for a sample of the Telegraph's science reporting, written by Adam Boult, who I suspect didn't complete his statistics course:

In a workshop over the weekend at the Annenberg Public Policy Center,  one of the presentations was based on a paper by Dan Kahan et al., "Culturally antagonistic memes and the Zika virus: an experimental test", Journal of Risk Research 2017. The abstract starts this way [emphasis added]:

This paper examines a remedy for a defect in existing accounts of public risk perceptions. The accounts in question feature two dynamics: the affect heuristic, which emphasizes the impact of visceral feelings on information processing; and the cultural cognition thesis, which describes the tendency of individuals to form beliefs that reflect and reinforce their group commitments. The defect is the failure of these two dynamics, when combined, to explain the peculiar selectivity of public risk controversies: despite their intensity and disruptiveness, such controversies occur less frequently than the affect heuristic and the cultural cognition thesis seem to predict. To account for this aspect of public risk perceptions, the paper describes a model that adds the phenomenon of culturally antagonistic memes – argumentative tropes that fuse positions on risk with contested visions of the best life. Arising adventitiously, antagonistic memes transform affect and cultural cognition from consensus-generating, truth-convergent influences on information processing into conflictual, identity-protective ones.

During the discussion, someone remarked in passing that these things are properly not memes or tropes but rather frames.  What follows is a bit of idle lexicographic investigation into this terminological tangle.

This is a guest post submitted by Nathan Sanders and colleagues. It's the text of an open letter to Neil deGrasse Tyson, who made a comment about linguists on Twitter not long ago.

I commented back in 2008 on the ridiculous vagueness of some of the brief weather forecast summaries on BBC radio ("pretty miserable by and large," and so on). I do sometimes miss the calm, scientific character of American weather forecasts, with their precise temperature range predictions and exact precipitation probabilities. In recent days, on BBC Radio 4's morning news magazine program, I have heard an official meteorologist guy from the weather center saying not just vague things like "a weather front trying to get in from the north Atlantic," or "heading for something a little bit warmer as we move toward the weekend," but (more than once) a total baffler: "The temperature is going to be struggling." What the hell is that about?

"Slow-talking the inaugural" was just reposted in Significance, a a statistics magazine published by the American Statistical Association and the Royal Statistical Society. Or following their logo,

which I guess can be approximated via Unicode as SIGNIFICΛNC≡.

Mark Ziemann, Yotam Eren and Assam El-Osta, "Gene name errors are widespread in the scientific literature", Genome Biology 2016:

The spreadsheet software Microsoft Excel, when used with default settings, is known to convert gene names to dates and floating-point numbers. A programmatic scan of leading genomics journals reveals that approximately one-fifth of papers with supplementary Excel gene lists contain erroneous gene name conversions.

Emily Hopkins, Deena Weisberg, and Jordan Taylor, "The seductive allure is a reductive allure: People prefer scientific explanations that contain logically irrelevant reductive information", Cognition 2016:

Previous work has found that people feel significantly more satisfied with explanations of psychological phenomena when those explanations contain neuroscience information — even when this information is entirely irrelevant to the logic of the explanations. This seductive allure effect was first demonstrated by Weisberg, Keil, Goodstein, Rawson, and Gray (2008), and has since been replicated several times (Fernandez-Duque, Evans, Christian, & Hodges, 2015; Minahan & Siedlecki, 2016; Rhodes, Rodriguez, & Shah, 2014; Weisberg, Taylor, & Hopkins, 2015). However, these studies only examined psychological phenomena. The current study thus investigated the generality of this effect and found that it occurs across several scientific disciplines whenever the explanations include reductive information: reference to smaller components or more fundamental processes. These data suggest that people have a general preference for reductive information, even when it is irrelevant to the logic of an explanation.

John Oliver on TV Science — featuring the TODD talks ("Trends, Observations, and Dangerous Drivel"):

Biologists are figuring out what many other fields learned decades ago:

What it feels like trying to publish. #ASAPbio pic.twitter.com/bJCaFqrop7

— Jonathan A. Michaels (@JonAMichaels) February 25, 2016

See Amy Harmon, "Handful of Biologists Went Rogue and Published Directly to Internet", NYT 3/15/2016. Also see "Reviewer Two must die".

Academic journals are on their way to playing the same role in the life of science and engineering that caps and gowns do: a quaint cultural relic that plays a role in celebratory rituals, but has nothing to do with the day-to-day process of exploration, discovery and communication.