Independent Labs To Verify High-Profile Research Papers 74
ananyo writes "Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results. The problem of irreproducible results has gained prominence in recent months. In March, a cancer researcher at Amgen pharmaceutical company reported that its scientists had repeated experiments in 53 'landmark' papers, but managed to confirm findings from only six of the studies. And last year, an internal survey at Bayer HealthCare found that inconsistencies between published findings and the company's own results caused delays or cancellations in about two-thirds of projects. Now, 'Reproducibility Initiative,' a commercial online portal is offering authors the chance of getting their results validated (albeit for a price). Once the validation studies are complete, the original authors will have the option of publishing the results in the open access journal PLoS ONE, linked to the original publication."
Re:cool! (Score:4, Insightful)
Negative results are sometimes just as interesting as positive ones. As you usually learn something.
You would think.
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
Re:cool! (Score:4, Informative)
Negative results are sometimes just as interesting as positive ones. As you usually learn something.
You would think.
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis. Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
I want to live on your planet [Re:cool!] (Score:4, Insightful)
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis.
That would be nice if things were that simple.
Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.
On my planet, stuff doesn't always work. When it doesn't work, it's not always easy to figure out why it doesn't work. When you don't get a result, it's hard to be confident that you didn't do something wrong-- and the people who are confident that they didn't do something wrong... are often wrong. It's not always trivial to say whether or not you did something wrong, or whether the experiment set up had a flaw, or there was something that turns out to be important that you didn't know was important, or whether the result you're trying to replicate was just wrong to start with.
Re: (Score:2)
In the ideal world, that would be true, but in the real world, what you most often learn is that there are many different ways to screw up a delicate measurement in ways from which you learn little or nothing.
What are you talking about? Negative results doesn't mean someone screwed up a measurement. Negative results means the experiment ran correctly but the results went counter to the hypothesis.
That would be nice if things were that simple.
Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.
On my planet, stuff doesn't always work. When it doesn't work, it's not always easy to figure out why it doesn't work. When you don't get a result, it's hard to be confident that you didn't do something wrong-- and the people who are confident that they didn't do something wrong... are often wrong. It's not always trivial to say whether or not you did something wrong, or whether the experiment set up had a flaw, or there was something that turns out to be important that you didn't know was important, or whether the result you're trying to replicate was just wrong to start with.
On my planet, I never said or implied that science was easy or trivial. Science IS hard, and if you are submitting to a journal without taking the hard steps of ensuring your measurements are accurate and your methods are not flawed, then you are contributing to bad science. I was not saying that every submitted experiment should be guaranteed to be flawless and come with a "label." I was saying that a scientific error in the experiment is not the same thing as a negative result. You can have scientifi
Re: (Score:2)
Science IS hard, and if you are submitting to a journal without taking the hard steps of ensuring your measurements are accurate and your methods are not flawed, then you are contributing to bad science.
Spoken like someone who never did any research, where a flaky cable is responsible for that super-interesting trend you discovered.... until someone points out the flaky cable. Most people do exactly what you recommend, and are tripped up by flaws in their measurements that they did not foresee or find.
Finally, when you respond to this comment, geoffrey.landis, try to act like an adult. There is no need to be a smart-ass, or glib, or whatever it is you were trying to be.
If you want to be taken seriously when talking about how science works, what scientists should do to get results and how scientists should behave when working on their experiments, you might want to figure ou
Re: (Score:2)
Spoken like someone who never did any research, where a flaky cable is responsible for that super-interesting trend you discovered.... until someone points out the flaky cable. Most people do exactly what you recommend, and are tripped up by flaws in their measurements that they did not foresee or find.
To a large extent, I would say that a scientist has the responsibility to ensure his tools are working correctly before publishing findings; and yes, that includes verifying that a cable is not flaky. Besides that, I agree that not every error can be accounted for or foreseen, which is exactly why I said that sort of thing is that the journalling review process is for. But all that is just distracting from the my main point which is that making errors in an experiment is not the same thing as getting nega
Re: (Score:2)
By claiming that negative results are worth as much as positive results.
Please enlighten me on where I said or implied anything even close to that statement. My only point was that mistakes did not equate negative results. Nowhere did I suggest that negative results and positive results had the same worth. You and GP seem determined to read more into my statements than what I am actually typing. You seem to be reading somewhere that I think science is somehow perfect and researchers are without flaw. I said no such thing. The closest I came to such as statement was when
Re: (Score:1)
Hey donaggie, don't take what they say to heart. Some people will twist and contort some minor detail in your comment and then run with it like it must be the only way to interpret what was typed. You (and the initial AC) are absolutely correct that much can be learned from a properly set up experiment that did not produce the result the scientist was hoping for. For instance, I work in a field where the chemical reactions aren't easily predictable. I mix two precursors because the evidence I have says it w
Re: (Score:2)
Re: (Score:1)
Negative results are the fruit of good science just as much as positive results are. Screwing up the measurements in an experiment is simply bad science, or not science at all.
What planet are you from? I want to move to your planet, where science is so easy, and stuff always works unless it's "bad science," which apparently comes with a label so anybody can tell which is which.
Good science is designed so that even 'negative' results are reportable and interesting. Within many fields (eg, biology) few experiments are really designed that way. Many experimental designs are simple, two-factor designs such as [normal-population disease-population] X [placebo real-drug]. The outcome of this is: "Yes, the drug does something" or "We couldn't measure a significant effect." Failure to measure a significant effect is different from a negative result. Failure to measure a significa
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
This has nothing to do with negative results. This is to weed out false positives.
Always been a problem (Score:5, Interesting)
That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done.
I'm puzzled by this sentence, though: "Once the validation studies are complete, the original authors will have the option of publishing the results in PLoS ONE, linked to the original publication. "
They're saying that the people who did the work replicating the experiment don't get the credit for their work, but instead the authors of the paper that they're trying to replicate do?
And, what if the new work doesn't replicate the results? Does it get published? Credited to whom?
Re: (Score:2)
"That's always been a problem; the journals usually want to publish new work, and aren't interested in publishing work that just repeats something already done."
That has been a problem as you say.
"And, what if the new work doesn't replicate the results? Does it get published?"
Well, that is already part of the previous problem, also since long.
Very few journals have been publishing "contradictory results", unless "seriously warranted", or something along those lines.
"Credited to whom?"
Yes, this is curious. W
Re: (Score:2)
I've never run into that. What usually happens is that researchers don't produce contradictory results, they produce inconclusive ones. As it is, lots of inconclusive studies get published with discussions and conclusions that imply they are negative findings.
Re:Always been a problem (Score:4, Informative)
Re: (Score:2)
It Thus to do something like this you need to assign one or more very capable senior students/postdoctoral workers, which costs money and time and takes away from original research.
As someone who gave up on research science in organic chemistry, due to the amount of faulty and incomplete synthesis procedures I encountered in the literature, not doing this costs money and time and takes away from original research. ("Oh yes. We've encountered difficulty getting a high yield in that step as well. We do this." "Um yeah, so why wasn't that in any of your published articles over the last four years?")
Re: (Score:3)
This is being discussed at ITP (Score:2)
http://pipeline.corante.com/archives/2012/08/14/reproducing_scientific_results_on_purpose.php [corante.com]
and
http://pipeline.corante.com/archives/2012/08/15/more_on_reproducing_scientific_results_organic_chemistry_edition.php [corante.com]
Re: (Score:2)
Obama isn't a marxist. He isn't even a socialist.
He is a moderately conservative pro corporate welfare capitalist.
Re: (Score:2)
You really are not worth replying to you anonymous coward. But I will inform you anyway.
Yes. Obama really is conservative.
The "American" definition of conservative is not conservative. It is a radical regressive movement attempting to overturn the liberties that we have fought and died for since the American and French revolutions. It is attempting to return to a dark age where a select class of gentry wields absolute authority and power through economic, political and superstitious means. In the past, this
How can we implement this in practice? (Score:4, Insightful)
The idea to reproduce important results is good and is part of the scientific method. In practice this is much harder to accomplish due to several constraints. I can only speak for my field but I think this applies to other fields as well that the reproduction is hard by itself.
This leads us to a bunch of problems. If it takes a graduate student a year to collect a data set on a custom made machine that is expensive and time consuming who has the resources to reproduce the results? In most branches we are limited by the available personnel. It is hard to imagine giving someone the task of 'just' reproducing someone else's result, as this does not generate high-impact publications nor can be used for grant applications.
The thought behind this would benefit the scientific progress, especially to weed out questionable results that can lead you far off track but someone needs to do it. And it better not be me, as I need time for my own research to publish new results. Any reviewer always asks him/herself whether this is really an achievement that it is worth publishing, which reviewer would accept a paper stating "We reproduced X and did not find any deviations from the already published results" ?
Fail [Re:How can we implement this in practice?] (Score:5, Insightful)
Yes, the more I look at this, the more I see little chance for it to work.
A graduate student will typically spend ~2 years of dedicated study in a narrowly specialized field to learning enough lab technique to do a difficult experiment, often either building their own custom-made equipment, or using one-of-a-kind equipment hand-built by a previous graduate student, and do so with absurdly low pay, in order to produce new results. You can't just buy that; they're working for peanuts only because they are looking for the credit for an original contribution to the field. And then you're going to say "oh, by the way, the original authors get the publication credit for your work if it reproduces their results, and we won't publish at all if you don't."
And, frankly, why would the original researchers go for this? You're really asking institutions to pay in order to set up a laboratory at a rival institution, and then spend time and effort painstakingly tutoring their rivals in technique so as to bring them up to the cutting edge of research? And even if you can bring a team from zero up to cutting-edge enough to duplicate your work, what you get out of it is a publication in a database saying you were right in the first place?
Re: (Score:2)
Re: (Score:2)
This is why the LHC has ATLAS and CMS as I understand it - different detectors looking at the same beamlines, with independent teams (at least as I understand it).
Dumb racket (Score:4, Informative)
Re: (Score:2)
Who will pay for this? (Score:4, Insightful)
Re: (Score:1)
They're hoping to establish their certification as important, trustworthy, etc. It sounds nice on paper but when you get down to it it's still for-pay research seeking a previously determined outcome, and another in the long line of ultimately pointless certifications/accreditations.
If they're successful, then anyone at a public institution who wants to be published will strive for that certification, and will demand essentially double the grant money.
Universities will pay for it because they're mired in a
Re: (Score:2)
It seems to me the right people are currently paying for replication. If a drug company wants to use some results, they replicate them first. The drug company SHOULD pay for that study. If someone else in academia is interested in using a result, they replicate it first.
The problem seems to be that people, including most researchers, put entirely too much faith in individual studies.
Well now (Score:4, Insightful)
I can see this happening for some fairly small studies, but many very big studies simply can't be replicated. For example, a big public health study will possibly change the sampling population. What about the LHC? How could anyone realistically replicate that work? The deal is replication isn't really replication as you can't always copy what someone has already done. This idea just seems more like profiteering than anything else. What we really need are options for research groups to publish studies that failed but say something interesting about why they failed. This is much more useful. This way we all learn. Plus big labs aren't always free from suspicion themselves.
Re: (Score:2)
I can see this happening for some fairly small studies, but many very big studies simply can't be replicated. For example, a big public health study will possibly change the sampling population.
Err...so? If the study results can't be supported across a different sampling population (that still meets the study's stated criteria), then the original study results should be revised or invalidated. In fact, it's a better acid test if they do change the population.
fictitious e.g.: if a study emerges that says eating X amount of tomatoes increases the chance of bearing twins in women ages 19-29, but changing the sampling population to people from a different country (still eating tomatoes, still 19-29
Re: (Score:1)
Ratios (Score:5, Interesting)
>53 'landmark' papers, but managed to confirm findings from only six of the studies.
That's 89 percent crap. (88.7%)
Sounds about right.
And as I get older, it seems that this observation holds true more every day, in everything.
--
BMO
Re: (Score:1)
Re: (Score:2)
> If scientists are afraid to publish risky results that have never been observed before, then scientific progress will slow down.
That's not what I'm advocating.
I'm just making the observation that Sturgeon's Law holds true here too. The overwhelming majority of stuff everywhere is mediocre at best. As for scientific studies, I think the problem is a result of publish-or-perish *and* a bias toward positive results, as opposed to negative results (we did this and it didn't work). As if coming up with
Re: (Score:1)
They need this for medical studies too (Score:1)
Multi-variable medical studies need something like this as well. They also need to have 'real world' results to see if their study findings scale to the millions of people in the general population.
College students should be the ones doing this type of stuff. Universities have a budget for this, it will teach the kids what is current in their field, and get them exposure to the test equipment and process.
Also, isn't this what peer-reviewed is supposed to be, prior to getting published?
The Price Could Be Interesting... (Score:2)
Easy for some fields (Score:2)
In all computer related fields, that's pretty easy: give the code. It's often a pain in the ass to reproduce the results (and I talk only for my field), but as soon as you get the code, then you see what's the tricky part, and what's left to improve.
Looks like they get something either way... (Score:2)
If there's confirmation, they get to republish their results. If there isn't, they get to republish in the JIR [jir.com]
Statistical confirmation (Score:3)
The costs involved in performing research would preclude this working in most fields. However where there would be considerable value in this sort of 'out of house' service is in performing re-analysis of the raw data behind the publication. Stats is hard and unfortunately it's all too easy to make a bit of a hash out of it. Unfortunately the current peer review process doesn't always address this adequately - either because the reviewers aren't neccessarily any better at statistics themselves or the data as presented has been stepped through processing that may add unexpected bias. Having a career statistician run a leery eye over the analysis in the orginal Wakefield paper [doi.org] certainly wouldn't have hurt.
Re: (Score:1)
Re: (Score:2)
True the results were faked in some cases but putting those aside (and going completely from memory) the groups were still poorly matched and the conclusions bore little relation to the analysis. It's more that I was getting at - do the data/statistics support the claim of the paper - and I don't think they did.
For catching intentional faking I've always quite liked Benford's Law [wikipedia.org] (according to a very short section in Wikipedia article it's been successfully tested against scientific fraud). At the very le
Very expensive (Score:2)
There have been many hallmark publications on cancer research, which usually involve at the very least extensive animals studies. Many involve human subjects. The cost to test drugs or theories on humans is extensive. Most scientists don't have the funds to redo these experiments, and wouldn't want to either -- the grant money they receive would be towards building on previous results. No funding agency would give money to re-verify a study already published.
At the very least, the authors could make ALL dat
Re: (Score:2)
The cost is irrelevant: if anyone is to build on your results, they should first look for confirmation.
Why?
- Publications suffer a huge selection bias -- it is nearly impossible to get a negative result published (even if it refutes prior publications!).
- Most statistical work (a) uses an absurdly generous 95% confidence interval and (b) hasn't even been vetted by a competent statistician.
Requiring data and code to be published would go a long way towards improving the situation, since there's no point in r
Question about getting published (Score:1)
Re: (Score:1)
Great Idea! (Score:2)
As a scientist I have to wonder (Score:2)
I like the idea, but, how do you fund this effort? I don't see the article making any mention of this.
We all spend our time writing grants now to support our own research and have little enough time to do it. Now we're expected to do someone else's research? I suppose it's a bit like reviewing articles, if you want to publish, you should review. However, this takes much more time, effort and money to do.