Forgot your password?
typodupeerror
Biotech Science

How Did Those STAP Stem Cell Papers Get Accepted In the First Place? 109

Posted by samzenpus
from the don't-press-send dept.
bmahersciwriter writes The news team at the scientific journal Nature turns its investigative power on the journal itself. The goal: to try and understand how two papers that made extraordinary claims about a new way to create stem cells managed to get published despite some obvious errors and a paucity of solid evidence. The saga behind these so-called STAP cells is engaging, but sadly reminiscent of so many other scientific controversies.
This discussion has been archived. No new comments can be posted.

How Did Those STAP Stem Cell Papers Get Accepted In the First Place?

Comments Filter:
  • "See, this is exactly why we oppose stem cell research. They are all frauds."
    Seriously though, I would have imagined that the papers should only get published if the results themselves were reproducible. Somehow those are skipped and the whole peer review system is in trouble. At the end, I would think whoever reviewed the papers should also be disciplined.
    • Re: (Score:3, Insightful)

      by Anonymous Coward

      And where would the money to reproduce the results come from? Journals typically do not even pay peer reviewers, so you're essentially expecting them to use their own grant money and time to reproduce results knowing that it will get them precisely fuck-all in terms of career advancement. If that were actually the expectation the simple reality is that it would kill off the peer review system.

    • by Anonymous Coward on Thursday July 03, 2014 @09:13PM (#47380989)

      Seriously though, I would have imagined that the papers should only get published if the results themselves were reproducible. Somehow those are skipped and the whole peer review system is in trouble.

      Actually, the whole peer review system is not in trouble. See, the peer review done by (volunteer) reviewers for the magazine is just the first step. The next step comes when the article is published and the entire world gets to see the paper. The fact that the fraud was exposed in pretty short order after publication shows that, indeed, peer review does work.

    • by pepty (1976012) on Thursday July 03, 2014 @11:18PM (#47381371)
      Until grants provide funding for reproducing old results, publishing a paper will remain the first step in getting the work reproduced. Typically by someone who needs the technique/result for their own work.
    • by evo2 (2022012)

      Seriously though, I would have imagined that the papers should only get published if the results themselves were reproducible.

      Who is going to reproduce unpublished work? Who is going to even know there is something reproduce if it hasn't been published? Publication of the original study necessarily comes before independent replication.

      Somehow those are skipped and the whole peer review system is in trouble. At the end, I would think whoever reviewed the papers should also be disciplined.

      I think you don't understand how peer review works. Publication is just the first step.

    • by aussersterne (212916) on Friday July 04, 2014 @02:19AM (#47381853) Homepage

      Journals aren't arbiters of Truth (capital T), they're just what they say they are: JOURNALS of the ongoing work of science.

      Someone records that they have done X in a journal. Because said journal is available to other scientists, other scientists get to try to make use of the same notes/information/processes. If they are able to do so, they journal it as well. Get enough mentions in enough journals that something works, and we can begin to presume that it does.

      If only one mention in one journal is ever made, then it is just another record in another journal of another thing that one scientist (or group of scientists) claim to have done.

      Peer review is just to keep journals from expanding to the point that there is too much for anyone to keep track of or read. It is emphatically NOT the place at which the factuality or truthfulness of notes/information/processes are established once and for all. That happens AFTER publication as other scientists get ahold of things and put them through their paces.

      Seriously, this is all exactly as it is supposed to work. I have no idea why there is such hoopla about this. There is nothing to see here. One group journaled something, other groups couldn't replicate it, they no doubt will reference this failure in future articles, and "what happened" is recorded out in the open for all of science, thereby expanding our pool of knowledge, both about what consistently works in many situations and of what someone claims has worked once in one situation but appears either not to work in the general case or requires more understanding and research.

      Again, there is nothing to see here. Let's move on.

    • by jmv (93421)

      So you're saying that reviewers should have to reproduce the results (using their own funds) of the authors before accepting the papers or risk being disciplined? Aside from ending up with zero reviewers, I don't see what this could possibly accomplish. Peer review is designed to catch mistakes, not fraud.

      • by thieh (3654731)
        I am saying the Journal should put up some funding to reproduce the results. It can be random testing or all the papers, but the journal should invest in keeping its own reputation of being fraud free, no?
        • by jmv (93421)

          It would certainly be nice, but it's not realistic. For a simple paper, it would likely cost a few thousands, but for anything that requires fancy material, it could easily run in the millions. The only level where fraud prevention makes sense is at the institution (company, lab, university) level.

    • by Reziac (43301) *

      I think it should be all right to publish something with the caveat, "This is what we found, but we couldn't reliably reproduce it. Let's have someone with new eyes take a look and see if we're on to something or full of shit."

      Because it's just as useful to know what *doesn't* reliably work, and you can't know about it unless it's documented.

      But publishing one-offs as reliable research precedents is, not to put too fine a gloss on it, lying.

  • I can't imagine this was done on purpose... I can't imagine how a scientist would, knowingly, publish wrong results (and perceived as revolutionary/important by their peers). Because this would be nothing more than willingly putting a sword of Damocles over your head / committing professional suicide on the spot. I mean, how is that possible that rational people (scientific minds) would accept to do such thing while being sure it will compromise their entire career (and life) after that?

    • by Anonymous Coward

      Hmm...well, without some people over time DOING EXACTLY THAT (putting out unorthodox ideas, throwing their careers or lives away because the orthodoxy wouldn't put up with it at the time) we'd still be believing/being told that the earth was flat, the center of the universe, etc.

    • by Beck_Neard (3612467) on Thursday July 03, 2014 @09:08PM (#47380967)

      It's probably not the case that she wrote the paper cackling to herself madly and proclaiming "Those suckers will never find out!"

      It's probably the case that through self-delusion and carelessness she managed to partially convince herself that the results were true, and this, coupled with pressure to produce results, caused her to take a few shortcuts to get it published. What she did was wrong, and her career is over. It's not something a rational mind would have done. But scientists are just human and sometimes prone to making irrational decisions. The great thing is that we have the scientific method to weed out the good from the bad.

      • by Anonymous Coward

        What she did was wrong, and her career is over.

        You don't seem to understand how Science works. Theories that later turn out to be wrong are proposed all the time, and doing so is not a blot on your academic record unless it is proved (or strongly believed) that you did it on purpose and hence wasted everybody's time knowingly .

        If it wasn't deliberate then making an erroneous proposal is a perfectly normal occurrence in Science, and we rely on other scientists to confirm or invalidate our work. Obviously

        • I think you've misread the post. In "What she did was wrong", I read "wrong" as "unethical", "unscientific", or at the very best "incompetent". Your criticism assumes it meant "something which eventually turned out not to be how reality works".

        • If you look at the case, there's evidence she did slight but deliberate manipulation and misrepresentation of results. That's what I meant by wrong.

      • by vossman77 (300689)

        Self-delusion seems to be extreme or unlikely in this case. Based on this blog [blogspot.com], she plagiarized her entire abstract only changing the cell name from ES to STAP.

        A simple plagiarism detector would have detected this fraud. I run turnitin.com service on all my students' papers and I catch plagiarizer every semester. I think this should be routine for high profile article going in to Nature or Science.

        • According to TFA, Nature did run it through a similar database. No obvious plagiarism was found. It turned out that the journal she originally published in was not in the database. There are one hell of a lot of journals out there.

        • Two questions: first, do you tell your students that you're going to be doing this? (I ask because it it would say quite a bit about the plagiarizers if they knew in advance that you'd be checking.) Second, how do they try to justify their actions when they're caught?
          • by vossman77 (300689)

            Yes, they can actually look at the plagiarized sections and fix them. The reaction tends to depend on how I contact them. If I am nice and say they can redo it for most the points, they admit their mistake and fix it. If you say they are getting a zero, then they start lying and say they do it in other classes and it is fine.

    • by gweihir (88907) on Friday July 04, 2014 @05:34AM (#47382319)

      One very common scenario for knowingly faked "results" is this: PhD Student has his/her funding running out and gets set an ultimatum (explicitly or implicitly). PhD student fakes something, sometimes looking pretty good at first glance. Advisor is too stupid, lazy or full of him/herself to notice. Paper gets published because advisor is "respected" in the community.

      I have seen this happening quite a few times, including one case where all authors, except the first one (the PhD student), wrote what was basically a retraction a year later. But did anything happen to these people? No. The PhD student still has his PhD, despite his results being essentially worthless. The other authors still have their reputation. The faked publications were not retracted. I did recognize the fake in 10 Minutes by numerous inconsistent things and numbers that did not add up and did not make sense at all. None of the reviewers apparently did. Just when people tried to reproduce the results and failed were some question asked. But as I said, no consequences for blatant scientific misconduct by several people. For me, this nearly cost me my PhD as my advisor was not even capable of understanding the fake after I explained it to him in detail and somehow though they were doing something vastly superior to my work. While the low-point of my scientific work, it made me understand that most so-called "scientists" do not qualify for that distinction.

      • by schnell (163007)

        I did recognize the fake in 10 Minutes by numerous inconsistent things and numbers that did not add up and did not make sense at all. None of the reviewers apparently did.

        This isn't intended to be disrespectful, so please don't take it the wrong way: why were you, as a PhD student, able to find this error when the reviewers (and theoretically other scientists in the field) weren't?

        Is yours a small field with few people to review? Were the reviewers of this paper lazy or cowed by celebrity or influence? Was this published in a seldom-read journal? Or what? I'm honestly very curious about how a lapse like this happens.

        • by gweihir (88907)

          I do not know what the reviews for the paper were. I only know it got into a well-known "Tier-I" conference. I do know my 10-Minute assessment was right, because more than a year later, the authors (minus the first one) had their follow-up paper where they basically admitted all defects and scientific misconduct by the first author. And I do know nothing happened to anyone. This was "mainstream-research", the conference is large and well-known.

  • "Nice journal you got here. Shame if it got broke..."

  • A reviewer might think that the results are not implausible, so the paper can go ahead. Reviewers don't try to reproduce the results in their own lab.
    • by Arkh89 (2870391)

      Reviewers don't try to reproduce the results in their own lab.

      Reviewers don't have human*time and financial resources to reproduce results in their own lab.

      FTFY

    • by Foske (144771) on Friday July 04, 2014 @03:36AM (#47382013)

      As a former reviewer working for a very renowned research institute in Europe I can say: Peers typically don't get/take the time to do their job right, and often outsource the job to less experienced people. Reproducing results is a very expensive and time consuming job, which means: unless it is it won't happen. You must be lucky if the reviewers have at least read the paper till the end. Quite often the review happens by people who are "no experts" in the field of the paper. For many conferences, papers with a bad rating still pass because there are not sufficient good papers, or if it is easy to guess the institute the authors work for, the paper passes without proper review.

      Once our institute had a paper rejected, but my boss -who was in the review team- managed to get the paper accepted anyway. High profile conference in Electronic Engineering.

      As a former paper author I can say: If your paper is rejected for one conference, you simply resubmit to another until it is accepted. Publish or perish is the holy grail of research, something many bosses will make very clear to you, and quality is less important. You don't write a paper because you have results, you write a paper because this or that major conference has a deadline in two weeks. I have a few paper on my name I am ashamed of: Omitting the bad results in the measurements, compare with competitors only on the features you know you would win because the comparison doesn't make sense at all, bragging about results which are very bad, but you hide that by not comparing to (avoiding any reference to) competitors which are better.

      As you might understand, I quit the job. I left research and never ever want to have anything to do with it anymore.

      • by gweihir (88907) on Friday July 04, 2014 @05:24AM (#47382289)

        There are some islands of honest and competent conferences and reviewers, but they are usually in not very well known events. All that is mainstream, "Tier-I" conferences and Journals are fundamentally corrupt. I mostly left research for the same reason, but I occasionally still publish something these days. The difference is that I publish if I have something good and interesting, not when some stupid research administrator thinks I should have more papers. And I publish in a venue where I respect the people running the conference even if that gives a lot less scientific "reputation".

  • ... by the journal itself... investigating itself... the result will be that they don't find any wrong doing in their own journal.

    And then the media will report "journal finds no wrong doing"... and then we can all go back to sleep.

    Right guys?

    The issue is not this one journal. Its a general lack of scrutiny in science itself. They are not being audited. The data is not being checked. The experiments are not being replicated.

    • by serviscope_minor (664417) on Friday July 04, 2014 @05:20AM (#47382275) Journal

      The issue is not this one journal. Its a general lack of scrutiny in science itself. They are not being audited. The data is not being checked. The experiments are not being replicated.

      I believe you are mistaken, or alternatively I have a different interpretation of events.

      No system is perfect, however the system works more or less as follows.

      Scientists do work.

      They are under pressure to publish because not unreasonably the people paying for the scientists like to see their money is going somewhere useful even if the metric is far from perfect. Anyway it generates strong pressure since science is up or out an unlike many other careers, if you don't get promoted, you're eventually fired.

      Work is submitted to a journal or conference. Note in HEP (high energy physics) people just shove ut up at xxx.lanl.gov and send it to a journal something as an afterthought to satisfy the previous point.

      Editors filter the papers. At low journlas almost every submission gets sent out to review. At very competitive journals, the editors strongly filter papers. It helps to know an editor to get your papers past this filter. Some journals (Nature) have paid editors. Most to not.

      It gets sent out to other scientist volunteers to review it. The review is really a check of reasonableness. Nobody expects the reviews to find deliberate fraud. That's not the point. They're not there to check for actual correctness, only reasonableness. In other words no one excpects them to replicate the results. The reviewers have not much time. Some less scrupulous ones farm it out to students to review. Some are mad as a sack of badgers. Some will insist you cire them more. Others may be able to check some of the results for correctness if they look suspicious (I've done this before when I believe I've spotted a glaring error I can test). In some very simple cases (e.g. Magnesium Diboride as a semiconductor) people are able to verify the results before publication.

      The previous two steps repeat until the paper is accepted or rejected.

      Now the important part starts.

      There are two main possibilities here:
      1. The paper is boring. In this case it willprobably get a few citiations bulking up someone's lit review and nothing more.
      2. The paper is interesting in which case people will try to build on the results.

      And orthogonally:
      A. The paper is right.
      B. The paper is wrong (or bad or fraudlent or etc).

      (in practice there is a continuum on both)

      Basically in the case of 1 it doesn't matter if the paper is right or wrong. It languishes in obscurity and the rightness or wrongness never ends up having any bearing on the state of the art and "scientific knowledge".

      In the case of 2, people will try to build on it and take it further and advance the state of the art because that's where progres is happening. And that's where the scrutiny starts. The more interesting the result, the more intense the scrutiny.

      For results of the magnitude of cold fusion and high temperature scrutiny, every scietist and his dog tries to take a look. The results are very quickly found out to be bad (cold fusion) or good (High Tc).

      This was a big result, and what you're seeing now is the scrutiny.

      The journal peer review process is merely there to provide a good place to find new results and stop the scientific world getting flooded in a mound of utter dross. It's the very first part of the filtering process.

      Once a result is public is can undergo scrutiny and generally this happens in the public eye with people posting comments, other papers, commenting on the internet and so on.

      Scientific results in practice undergo more scrutiny than almost anything else, and you're seeing a little bit of it right here even on slashdot.

      • by Karmashock (2415832) on Friday July 04, 2014 @05:48AM (#47382355)

        Well, then whether something makes it through peer review or not should not be used as evidence that it is correct or not... merely interesting and reasonable.

        • Well, then whether something makes it through peer review or not should not be used as evidence that it is correct or not... merely interesting and reasonable.

          More or less yes. It's a weak piece of evidence in favour but it's very context dependent. Certainly anything new and contraversial should be treated with care.

          • I'll point out here that there is a lot of controversial science that is backed up almost entirely by its ability to get published.

            Much of the science is not reproduced and much of it doesn't include enough information to even attempt to reproduce it.

            that is something that I think would be a reasonable requirement for publication. Provide a full step by step break down of how you go from raw data to verified conclusion.

            I've read a few of these papers and they don't ever seem to do that.

  • by gweihir (88907) on Friday July 04, 2014 @05:21AM (#47382277)

    Peer review works if the people doing the review are honest and competent. Both aspects have been in sharp decline, not only in the biomedical field. These days, positions for Professors and PhD students are more often than not filled with people that can simulate competence and that have no or little personal ethics whatsoever. They will form groups that accept any and all papers from each other and reject anything from others. Anything original also generally has a high chance of getting rejected, unless the reviewers know and like the authors. The peer-review system is so broken and corrupt that it has just stopped working as the quality of the "researchers" forming it is way too often abysmally bad. (And forget about "anonymous reviews". The in-group has all the Tech-Reports from their friends and can recognize all papers written by them.)

    This is not a new phenomenon, it seems to just be getting worse again. But remember that Shannon had trouble publishing his "Theory of Information", because no reviewer understood it or was willing to invest time for something new.

    • by jmv (93421)

      This is not a new phenomenon, it seems to just be getting worse again. But remember that Shannon had trouble publishing his "Theory of Information", because no reviewer understood it or was willing to invest time for something new.

      That's the problem here. Should the review system "accept the paper unless it's provably broken" or "reject the paper unless it's provably correct". The former leads to all these issues of false stuff in medical journals and climate research, while the latter leads to good research (like the Shannon example) not being published. This needs to be more than just binary. Personally I prefer to accept if it looks like it could be a good idea, even if some parts may be broken. Then again I don't work on controve

      • by gweihir (88907)

        I think what is missing is that a) more reviewer actually need to be experts and practicing scientists and b) doing good reviews needs to get you scientific reputation rewards. At the moment,investing time in reviewing well is a losing game for those doing it.

        I agree that good reviews do not need to be binary. You can also "accept if this is fixed", "rewrite as an 'idea' paper", "publish in a different field", "make it a poster", etc. But all that takes time and real understanding.

        • by jmv (93421)

          I think what is missing is that a) more reviewer actually need to be experts and practicing scientists and b) doing good reviews needs to get you scientific reputation rewards. At the moment,investing time in reviewing well is a losing game for those doing it.

          Well, there's also the thing that one of the most fundamental assumption you have to make while reviewing is that the author's acting in good faith. It's really hard to review anything otherwise (we're scientists, not a sort of police)

          I agree that good reviews do not need to be binary. You can also "accept if this is fixed", "rewrite as an 'idea' paper", "publish in a different field", "make it a poster", etc. But all that takes time and real understanding.

          It goes beyond just that. I should have said "multi-dimensional" maybe. In many cases, I want to say "publish this article because the idea is good, despite the implementation being flawed". In other cases, you might want to say "this is technically correct, but boring". In th

Slowly and surely the unix crept up on the Nintendo user ...

Working...