Pitbull reviewers: threat or menace?

« previous post | next post »

Worth reading: Virginia Walbot, "Are we training pit bulls to review our manuscripts?", Journal of Biology 8:24, 2009.

I strongly support what Prof. Walbot has to say. One way to think about it: she's talking about the difference between the standards you ought to apply in order to believe something, and the standards you ought to apply in order to publish it.

Unfortunately, pit-bull reviewers are roughly balanced by cream-puffs, so that the refereed literature is simultaneously polluted by junk and bereft of some worthwhile stuff. Some glimmers of what might be a better system can be seen here and there on the web — arXiv.org is the best known — but given the difficulty of the problem and the cultural conservatism of the academy, change is likely be slow.

[Update: Peter McBurney has reminded me of a classic paper by J. M. Chambers and Agnes M. Herzberg ("A Note on the Game of Refereeing", Applied Statistics 17(3): 260-263, 1968). Here the authors introduce their model:

]



5 Comments

  1. marie-lucie said,

    March 16, 2009 @ 8:27 am

    This problem was addressed a year or two ago in an article in the Canadian academic periodical University Affairs. One anecdote cited a review editor who had to drop a reviewer from his list for starting his reviews with "This is the worst paper I have ever read".

  2. Mary Kuhner said,

    March 16, 2009 @ 2:35 pm

    I once received, in place of the expected two reviews on a paper, one (reasonably favorable) review and a note from the journal editor which said, roughly, "The second reviewer was apparently hostile to the concept of statistical inference; as the review was not useful, I am sparing you the annoyance of seeing it."

    But most of my scientific review experiences have been okay, and some have been excellent; and in retrospect I even appreciate the reviewer who said, "I know there' s a concept of 'least publishable unit' but this paper falls short even of that" because, alas, he or she was entirely correct.

  3. peter said,

    March 17, 2009 @ 2:41 am

    The economist and mathematician Graciela Chichilnisky at Columbia has written of her adverse experiences with editors and referees in those two fields, here:

    http://www.chichilnisky.com/pdfs/papers/187.pdf

  4. Eric Baković said,

    March 17, 2009 @ 2:43 pm

    Nature has a blog dedicated to the peer review process, which I found out about via this editorial on mentoring in the 14 June 2007 issue of the journal.

  5. peter mcburney said,

    March 22, 2009 @ 8:06 am

    Another game-theoretic treatment of academic publishing is in this (serious) paper:

    J. R. Faria [2005]: "The game academics play: editors versus authors", Bulletin of Economic Research, 57 (1): 1-12.

RSS feed for comments on this post