Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Science

Rude Paper Reviews Are Pervasive and Sometimes Harmful, Study Finds (sciencemag.org) 135

Posted by msmash from the closer-look dept.
sciencehabit writes: There's a running joke in academia about Reviewer 2. That's the reviewer that doesn't bother to read the manuscript a journal has sent out for evaluation for possible publication, offers condescending or outright offensive comments, and -- of course -- urges the irrelevant citation of their own work. Such unprofessional conduct is so pervasive there's even a whole Facebook group, more than 25,000 members strong, named "Reviewer 2 Must Be Stopped!" But it is no laughing matter, concludes a new study that finds boorish reviewer comments can have serious negative impacts, especially on authors belonging to marginalized groups.

The study surveyed 1106 scientists from 46 countries and 14 disciplines. More than half of the respondents -- who were promised anonymity -- reported receiving at least one "unprofessional" review, and a majority of those said they had received multiple problematic comments. Those comments tended to personally target a scientist, lack constructive criticism, or were just unnecessarily harsh or cruel, the authors report. For example, one author received a review that stated: "The phrases I have so far avoided using in this review are 'lipstick on a pig' and 'bullshit baffles brains.'" Another reported receiving this missive: "The author's last name sounds Spanish. I didn't read the manuscript because I'm sure it's full of bad English."
This discussion has been archived. No new comments can be posted.

Rude Paper Reviews Are Pervasive and Sometimes Harmful, Study Finds More

Rude Paper Reviews Are Pervasive and Sometimes Harmful, Study Finds

Comments Filter:

  • I'm not reading TFA (Score:5, Funny)

    by rldp ( 6381096 ) on Thursday December 12, 2019 @02:49PM (#59513314)
    It was without a doubt written by a crybaby.

  • Describes my first day on /. (Score:3)

    by CaptainDork ( 3678879 ) on Thursday December 12, 2019 @02:56PM (#59513340)

    That's the reviewer that doesn't bother to read the manuscript a journal has sent out for evaluation for possible publication, offers condescending or outright offensive comments, and -- of course -- urges the irrelevant citation of their own work.

  • Reputations of reviewers? (Score:5, Insightful)

    by Anonymous Coward on Thursday December 12, 2019 @02:58PM (#59513352)

    Within a given journal, I would expect editors to informally or even formally give reviewers "reputation scores."

    While I wouldn't necessarily expect these reputations to be routinely shared with other journals, I wouldn't be surprised if some of the "worst" offenders' names circulated informally among editors of journals that covered the same field.

    For example, if I'm an editor and I get several substantiated complaints about "unprofessional reviews" from a given reviewer, I may quietly blacklist him from further reviews or quietly discard reviews he sends in. If I'm asked by another journal editor "do you have any recommendations for people to review for my journal" I won't give him that name, if he asks "what do you think about such and so" I will give the least-supporting answer or "non-answer" I can without getting myself or my employer into legal trouble.

    • Re:Reputations of reviewers? (Score:4, Insightful)

      by UncleGizmo ( 462001 ) on Thursday December 12, 2019 @03:07PM (#59513374)

      I was going to ask a similar question. Wouldn't editors know the identities of reviewers, and periodically "metamoderate"?

      • Re: (Score:3)

        by iroll ( 717924 )

        Editors know the identities of the reviewers and read the reviews.

        This is, at least, the case for traditional journals that require three qualified reviewers. Paper mills like MDPI may only require two reviewers; allow the reviewers to self-certify (basically any blowhard can review literally anything... published a paper in biology in the last 20 years? sure, you're invited to review a computer science paper); have editors that do little more than check boxes; and generally publish the paper after two roun

        • As opposed to citation mills where they'll have as many rounds as it takes to get the submission sufficiently fellatory to previous published works whether or not they're relevant. Assuming, of course, they don't arbitrarily reject you simply to prop up the artificial scarcity their reputation is predicated on.

          • Re: (Score:3)

            by ceoyoyo ( 59147 )

            Meh. Good journals will usually go as many rounds as required as well, despite their policies. The only difference is that if *all* the initial reviews are poor, a good journal will reject you.

            If you get the usual, two good reviewers and one idiot, the editor will usually let you argue with the idiot ad nauseam.

            The alternative would be for the editor to actually make a decision, and they *really* hate doing that.

          • Re:Reputations of reviewers? (Score:4, Interesting)

            by iroll ( 717924 ) on Thursday December 12, 2019 @06:22PM (#59514192) Homepage

            That, again, comes down to the editor. A competent editor at a reputable journal will not let an abusive editor spoil the show (I've powered past a few). Authors' response to comments are their chance to demonstrate to the editor that they understand a reviewer's concerns but do not agree with them.

            To be honest, it's better for papers to be rejected for thin reasons than accepted without criticism. A rejection (fair or unfair) isn't the end of the world; there's always other journals, and good journals let you flag people that you want excluded from reviewing your papers for this very reason.

            • Yes indeed, this is very surprising. I am often involved in peer reviews of scientific papers, and my experience does not relate to the summary of the article... How could a review with the comments shown in the article summary be vetted by the editor ? I've seen lazy reviews, I've seen nitpicking reviews, I've seen scientifically questionnable reviews, but I've never seen the gravel such as the ones shown in the article summary.
        • Damn. That's a cool site. I didn't even know about mdpi.com until your disparaging comment.
          Thanks iroll!
          ---
          "A lie told often enough becomes the truth." - Attributed at various times to either Joseph Goebbels or Vladimir Lenin.

    • Ah! But since we want to eliminate publishers there's no such safety device to protect authors. Sink or swim I say.

    • Re:Reputations of reviewers? (Score:4, Insightful)

      by jythie ( 914043 ) on Thursday December 12, 2019 @04:09PM (#59513654)
      One of the annoying parts though is that problematic reviewers also tend to have a lot of social capital. They are asses because their reputation allows them to get away with it and they are typically asses in ways that quietly resonate with enough of the editors and reviewers that they are willing to downplay it.

      • Re:Reputations of reviewers? (Score:5, Insightful)

        by Dixie_Flatline ( 5077 ) <<moc.liamg> <ta> <hog.naj.tnecniv>> on Thursday December 12, 2019 @06:26PM (#59514212) Homepage

        It's reputation and domain expertise. But I'm of the opinion that it doesn't matter how much of an expert you are in a topic, if you're unwilling to constructively participate in the scientific process, you shouldn't be invited to review papers. Even accounting for social capital, why they keep these people on is beyond me; they just slow everything down.

        • Re: (Score:2)

          by jythie ( 914043 )
          The problem is that they slow things down in a way that either does not affect people like them or even resonate with professional prejudices of a community and thus people like them do not see the slow down as a negative.

          For instance, I work AI research. We do Agent Based Modeling, and older and more expensive technique that lacks the 'only answers matter' results of modern deep learning techniques, but results are actually explainable.. meaning you can open up the model and explain WHY the agent did som

          • Oh man, my partner also works in machine learning, and she gets comments that are similar to that all the time. It's crazy, but everyone seems to have a silo and they refuse to consider solutions that don't go in their silo, regardless of whether or not it's the best solution for the problem. They''re a convolutional neural network expert, and the paper is on feed forward networks or whatever, they'll DEMAND to know why you didn't use a CNN for solving the problem. They absolutely refuse to believe that som

            • Re: (Score:2)

              by jythie ( 914043 )
              I think this is one of the downsides of lots of grant money and industry interest flooding in can be. Scientists usually have their silos, but other people's usually are not actually a threat. But once the money gets really good and you start seeing small numbers of really well funded labs spring up it starts becoming important that people see YOUR silo as the right one to bet on.

              In our case, we eventually lost our lab space to a group who was bringing in so much cash that their hardware purchases every

    • Re:Reputations of reviewers? (Score:4, Informative)

      by ceoyoyo ( 59147 ) on Thursday December 12, 2019 @06:14PM (#59514168)

      The secret is, editors are generally volunteers, overworked, and don't really care that much. They're happy if they can just get enough reviewers (most of whom were recommended by the authors themselves).

      Editors also will almost never take an actual stand. They want unanimous approval by reviewers, even if one reviewer is clearly an idiot (and the other reviewers agree with that assessment).

  • Conspiracies (Score:4, Insightful)

    by meerling ( 1487879 ) on Thursday December 12, 2019 @02:59PM (#59513354)
    And some fools think this fractious lot (scientists), are all involved in a world wide multigenerational conspiracy to fool the rest of mankind.
    Heck, they're lucky they can coordinate enough to schedule a conference that doesn't end in a bloodbath, or at least some really nasty bruised and walk-by-fruitings.

    • Re: (Score:2)

      by Shotgun ( 30919 )

      There are conspiracies, where a group agrees on a goal, then there are conspiracies of interests, where a goal has agreed on a group.

      -The deep staters in the CIA agreeing to create the Russia Hoax is a conspiracy.

      -German citizens agreeing to look the other way when it came to concentration camps is a conspiracy of interests.

      • Re: (Score:1)

        by HiThere ( 15173 )

        Sorry, but you're changing the definition.

        Conspiracy was literally "breathing together", and referred to people who met in groups that were small and close. As the definition widened, the groups were allowed to become larger, but intercommunication is the required link. In criminal law I believe it's required to be accompanied by at least one overt act.

        Thus if you wanted to call a labor union a conspiracy you'd have reason. But when you want to call a bunch of people who just kept the head down a conspir

      • German citizens agreeing to look the other way when it came to concentration camps is a conspiracy of interests.

        If a German citizen spoke up they would be murdered.
        If a climate scientists blew the lid off of the "hoax" they would win a Nobel Prize in Physics and become a celebrity scientist of world renown.

  • From TFA, one author received a review that "attack[ed] her ability to write in English."

    It is critical, that scientific papers be able to communicate. In most cases this means writing fluently in the language of the reader.

    If the publication's grammar really was poor enough to confuse the reader or even make the reader have to stop and think "um, what exactly is the author trying to say here" because of a lack of language proficiency, then it's very appropriate for a reviewer to point this out. However,

    • Re:Lack of English or other language proficiency (Score:5, Interesting)

      by jeff4747 ( 256583 ) on Thursday December 12, 2019 @03:33PM (#59513494)

      From TFA, one author received a review that "attack[ed] her ability to write in English."

      It is critical, that scientific papers be able to communicate. In most cases this means writing fluently in the language of the reader.

      It is also critical to read the thing you are commenting on, be it a scientific paper or a Slashdot article.

      For example, you decided to lecture based on this sentence:

      I didn't read the manuscript because I'm sure it's full of bad English."

      Utterly ignoring the first phrase so that you could pontificate about the second phrase.

      Perhaps you could spend long enough to actually comprehend the first phrase, and thus stop looking like a fool.

      • Re: (Score:1)

        by davidwr ( 791652 )

        For example, you decided to lecture based on this sentence:

        I didn't read the manuscript because I'm sure it's full of bad English."

        While that sentence might have gotten my attention, my " "lecture" [slashdot.org] was about the sentence in TFA, about an author who "wrote [an essay published in Science] about receiving a particularly negative review, on the first paper she submitted to a journal, attacking her ability to write in English."

        You do make a valid point - had I been responding to the author - not the one I wrote about but a different author - who received the "I didn't read the manuscript because I'm sure it's full of bad English" review, I w

  • Slashdot commenters are at least 100x worse than Reviewer 2. They will also think that posting about news is the same thing as peer review in a journal.
    I'm getting my popcorn and beverage to watch this one...

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      The difference is that you argue with idiots on Slashdot for recreation. You argue with idiot reviewers professionally.

  • ... especially on authors belonging to marginalized groups.

    World To End Tomorrow: Women, Minorities Hardest Hit.

  • Reviewing papers is one of those things that every student aiming for an "academic career" should do at least once before the end of the first year of graduate school, either "for real" or, more likely, in a "practice environment."

    Those same students should also go through the entire "paper submission" process by the end of their first year in grad school, either through publishing and going through the entire publication-and-review process, or doing so in a "practice environment." If they did publish but w

    • Substitute "professional forum poster" for "academic career" and you're almost there.

    • Reviewing papers is one of those things that every student aiming for an "academic career" should do at least once before the end of the first year of graduate school, either "for real" or, more likely, in a "practice environment."

      No absolutely not for real. First year grad students have no business reviewing papers. Many lazy profs give papers to massively unqualified students and that is a big part of the problem.

  • And I've never even heard of this being a thing. I've never had a belittling comment on a paper I've submitted. J. Rev. Phys., Phys. Rev A and B, J. Chem Phys., J. Opt. Instr. - all completely professional feedback. Where are these people publishing? The Inquirer?

    • Try publishing one to Slashdot.

    • Re: (Score:3)

      by ceoyoyo ( 59147 )

      I once had an IEEE TSP reviewer point out in his review that the Fourier transform does not, in fact, transform between the time and frequency domains. I specifically went to the library to locate a first year textbook to cite in rebuttal.

      I had a Lancet reviewer tell me that a particular algorithm could not be used in the way in which I had used it. I provided a literature search showing over 20,000 papers from the last thirty years that had used it exactly I had.

      The problem is certainly worse when you're d

  • I think the only insight being offered is that trolls have pretty much infested every angle of human life. Even deep down in the lofty academic world, there are trolls.

    Or that even in the ivory tower, you can't escape them.

  • Rules of thumb for useful criticism (Score:5, Insightful)

    by Tablizer ( 95088 ) on Thursday December 12, 2019 @04:55PM (#59513888) Journal

    1. Avoid overly general summaries, such as "this sucks". Let the details speak for themselves.

    2. Avoid vague criticism as much a possible. The vaguer it is, the less helpful it is. If you can't be specific, then you perhaps shouldn't be reviewing. If you have trouble articulating, take a break to ponder better wording. Sleep on it or work on something else. Most writing is better in general if you review it several days later.

    3. Keep criticism matter-of-fact, not decorated with negative words. Look for more pleasant ways to say the same thing. Make a personal catalog of reusable pleasant-leaning phrases for common categories such as pointing out contradictions, ambiguities, poor logic, etc.

    4. When pointing out that a statement is unclear or ambiguous, give examples of multiple possible interpretations. This helps the writer understand how your mind is (mis) interpreting their words. Example: "This phrase doesn't make clear whether the test-tube itself turned red, the contents inside it turned red, or both. The pronoun at X is not giving me enough info to tell and I didn't find other clues."

    5. Always apply Hanlon's razor, even if you do strongly suspect ill intentions. Accusing people of bad intentions makes problems worse 99% of the time. Focus on fixing content, not people.

    6. Don't comment when you are in a bad mood. Cheer yourself up first. Maybe a burger will help; full people are happy people.

    These are good life & office skills in general. Becoming a better critic is just a bonus. (Except for maybe the burger part.)

    • I would put this at the top:

      Go in with the intention of making the paper better.

      Even if the submission is ultimately irretrievable—which does happen, for any number of reasons—starting in good faith makes a big difference to the quality of the review. Starting with the wrong attitude ultimately makes each of your points effectively impossible.

      I'd hazard to say that most of the people being called out by this article are actually not putting in an effort in the way I describe, and it shows becaus

  • Buzzword bingo, it is. --We have "problematic", "marginalized", and more zingers, pretty clear that the author is looking to complain about people saying not-nice things to them and is trying to frame this as a problem that affects more than themselves. Criticism is built into the review process; it's not a bug, it's a feature, regardless of how harsh it may be.

  • and sometimes harmful, study finds

    Let's make this dangerous and harmful speech illegal [pbs.org] too!

Slashdot Top Deals

I tell them to turn to the study of mathematics, for it is only there that they might escape the lusts of the flesh. -- Thomas Mann, "The Magic Mountain"

Close