Peer Review Highly Sensitive To Poor Refereeing 233
$RANDOMLUSER writes "A new study described at Physicsworld.com claims that a small percentage of shoddy or self-interested referees can have a drastic effect on published article quality. The research shows that article quality can drop as much as one standard deviation when just 10% of referees do not behave 'correctly.' At high levels of self-serving or random behavior, 'the peer-review system will not perform much better than by accepting papers by throwing (an unbiased) coin.' The model also includes calculations for 'friendship networks' (nepotism) between authors and reviewers. The original paper, by a pair of complex systems researchers, is available at arXiv.org. No word on when we can expect it to be peer reviewed."
This reminds me of something else (Score:3, Interesting)
I can't quite remember what it was, but I seem to remember seeing it everywhere. It was exactly like TFA article, though. [wikipedia.org] Damn, what was that place called again?
The climate skeptics will have a field day (Score:4, Interesting)
This is precisely what the global warming skeptics say is happening with the global warming alarmist community. ie. scientists review each others' papers, in a 'co-operative' manner as it were.
I think I'll point some skeptics at this paper and then sit back with a bowl of popcorn and watch what happens.
Re:Highly political subjects? (Score:3, Interesting)
>>>a climate scientist who's an openly devout Christian finds data that sheds doubt on human caused global warming will be rejected because someone's afraid of looking foolish.
Nothing that extreme. More like they would reject papers that claim "global warming caused by natural causes" and accept papers that say "global warming caused by man", in order to protect their Own beliefs. A guy named Thomas Kuhn wrote about this very phenomenon (protecting the current paradigm aka worldview) several decades ago, about why the particle theory of light was initially rejected in favor of the existing "light passes through a medium" theory.
Basically the scientists/reviewers rejected papers as "hogwash" simply because they don't fit the accepted scientific theory. It can be a real challenge for new ideas to overcome this censorship.
Not to mention "autarkistic" research communities (Score:3, Interesting)
I mean scientists who publish among themselves, i.e. inside their narrow specialty, in their own journals, without checking whether the problem at hand has been solved elsewhere. This is more and more common as people get more specialized, and can lead to very basic errors propagated inside the whole community, like rheologists believing in the existence of pure elongational flow (a trivial misunderstanding of tensor algebra). Since the peers reviewing the papers are members of the same community, those errors usually get unnoticed.
It's part of automating the process. (Score:5, Interesting)
Just this week, I was asked to peer review a paper in which I was mentioned in the Acknowledgments. The request was sent out automatically -- the journal has records of all their authors, and the keywords for this paper matched the keywords in my profile, so I was picked to review it.
I recused myself, but really I should never have been asked. If they're going to handle the peer review process automatically, the artificial intelligence that makes the decisions needs to be improved.
The Social Text Affair (Score:3, Interesting)
The great anecdote demeaning peer-reviewed journals is The Social Text Affair [nyu.edu], where a prominent peer-reviewed journal published with enthusiasm the article "Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity", only to be informed it was, in fact, computer-generated gibberish submitted as a joke.
Important limitations in the Model... (Score:3, Interesting)
There are a couple of significant and important limitations in the model:
a) It assumes only two reviews per paper, and that the reviews are pure boolean, and that reviewer types are also pure and reviewers are randomly selected (when two of the classes of reviewers, 'mythantropes' (always reject) and 'altruists' (always accept) are specifically selected against by editors and PC chairs based on reputation).
b) It does not consider the cases (such as conferences) where there is a program committee meeting and the papers are not just considered on their own, but gone through a relative ranking process.
Re:Review content matters (Score:5, Interesting)
Regardless of all this though, sometimes you'll find out that only two of three reviewers responded, and at least one of those probably got one of their postdocs or even a PhD student to do the review. Some reviews will have empty parts where a reviewer was supposed to write a paragraph but couldn't be bothered, or because they didn't want to reveal the fact that they were totally unfamiliar with the subject matter. Getting a journal paper published is more hit and miss than you'd think. I used to think that a good paper with good ideas was enough, but it's not always the case.
Re:Just like the Slashdot moderation system (Score:5, Interesting)
Re:This reminds me of something else (Score:3, Interesting)
Wikipedia is completely different. There, you submit your work in whole and anonymous "referees" proceed to secularly mutate your effort into an intellectual monstrosity of its former self. Nothing is sacred. Some ignoramus actually removed all chemical equations from the Smelting article [wikipedia.org]. At least in peer review, referees simply make suggestions which you yourself implement.
Doing something unprecedented (Score:5, Interesting)
I actually read the comments on TFA, and down at , there's a particularly interesting one: [physicsworld.com]
This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.
There's actually a reasonably intelligent discussion going on in there...
Using Professors' Egos to your advantage... (Score:5, Interesting)
Re:Highly political subjects? (Score:3, Interesting)
I remember hearing about a nutrition paper that was rejected from a medical journal for a reason along the lines of "That can't possibly be true." So the guy updated the paper with an explanation of the basic bodily functions involved and how they work, which shows exactly why it could happen, and still rejected. He submitted it to a different paper where they basically said "This looks sound, we'll publish on the condition that you remove the explanation. Any doctor would know this already." The paper didn't fit in with the first reviewer's beliefs on nutrition, so he rejected it outright. A journal that was less biased on the subject approved it.
Now imagine that your research goes against the beliefs of all the referees. How do you get published then?
That same guy figured out how to get all his papers accepted without fail: simply load it down with so much math that the referee won't want to take the time to check your work. Since they can't find anything wrong with it because they didn't do the math, it's an automatic pass almost every time (except for cases like the above).
The issue applies to more than papers (Score:3, Interesting)
The government uses peer review to evaluate proposals for science and engineering grants. The same issues probably apply to those evaluations.
I have experienced a situation in which one reviewer recommended turning down a grant for reasons that could be considered as biased, although the bias was groupthink rather than individual. The other reviewers were enthusiastic about funding the grant and regarded it as a potential game-changer. It didn't get funded. A few years later the game-changing nature of the technology was recognized, but it was too late for the original applicant.
Re:Highly political subjects? (Score:5, Interesting)
As a computer scientist, my impression is that the program committees really are pretty random, or at least based on some sort of preference other than a widely agreed "quality" standard. Try it sometime: resubmit a paper rejected from a top CS conference verbatim to another top CS conference. The correlation between the reviews is usually quite low, both in terms of the numerical scores, and especially in terms of what they liked / complained about.
Re:just like /.? (Score:5, Interesting)
I've long thought there should be a "-1, Disagree" option in the drop-down box that takes a mod point but has no effect.
Re:just like /.? (Score:4, Interesting)
But the proper way to communicate disagreement should be to respond with a reasoned counterargument, the goal being to show the justification for your disagreement and allow future readers the benefit of seeing your reasoning.
A "-1 Disagree" mod is anonymous censorship at it's worst. It adds nothing to the discourse. If all that you can add to the discussion is "I Disagree", then you can't add anything of merit to the discussion.
Re:You mean whine when a POS paper is printed (Score:5, Interesting)
Contains such howlers as "There's no such thing as average temperature"...
Why is that a howler? It seems like uncontroversial physics if by "no such thing" you mean it has no physical meaning.
If I take two bodies, one with temperature T1 and one with temperature T2, what is their average temperature? If you say (T1+T2)/2 you are mathematically correct, but thermodynamically incoherent.
There is in general no thermodynamically meaningful way of averaging temperatures, which is why we should be talking about atmospheric heat content, not temperature, and why ocean temperatures--which are rising--are by far the most plausible evidence for increasing heat content in the troposphere.
Re:Highly political subjects? (Score:4, Interesting)
The poor review assignment at large conferences contributes to that effect as well, I think. I almost always have at least one of three reviewers, and sometimes even two of three, give a noncommittal review along the lines of, "well this isn't really my area, but it seems pretty good". Those reviews basically are non-reviews, so the acceptance decision is then entirely up to the remaining one or two reviewers. So it often comes down to: did the one person who actually provided an opinion on your paper like it or not like it?
In my experience that's often pretty subjective, especially for conferences with tight length limits (standard in AI is six pages). If the reviewer personally found the paper to be on an interesting subject with an interesting approach that he/she felt should be investigated, almost any shortcomings can be excused, and the reviewer will conclude that "Overall, this paper provides a valuable contribution to an important ongoing discussion in this area." But if the reviewer doesn't like it, finds it boring, dislikes the approach, etc., it's easy to find something that had insufficient detail, didn't sufficiently distinguish from related work, didn't sufficiently motivate the problem or investigate/validate the applications, etc., etc., since you really can't fit that much in six pages.
Re:just like /.? (Score:3, Interesting)
Sometimes after you've already used a couple mod points in a thread, you come across something that's +5 Insightful, yet you know is maliciously incorrect. People are replying to it as if the person was correct (they're +5 insightful must be right).
What do you do? Some people get mod points more than others... if someone only gets five points once every other month or so, at best, are they going to throw them all away by commenting in the thread? What chance does their comment even have of getting read, since a few people have replied already? Better to mod down and hope that either other people with mod points or people who are commenting pick up on it and realize the guy's wrong.
On the other hand, this is a great situation if other people who have replied to the +5 insightful yet wrong guy also realize the guy's wrong, because then you can mod those people up. But it doesn't always work that way.
I'm not saying it's the right way to use the system, but that's a common situation. You sometimes see people say "I'm giving up my mod points because you're so wrong", but you have to imagine most of the time they just down-mod instead. I often choose to use mod points in threads on a subject I know about rather than something I don't, which makes sense, but those are the threads where I'm most likely to have something to say as well; it's a bit of a conflict. I have some mod points now and was thinking of modding up a couple posts in here, but decided to respond to you... and the article is "old" enough that I'll probably not get modded up (or even read by anyone) so it's a wasted opportunity ;)
Re:The climate skeptics will have a field day (Score:2, Interesting)