How To Better Verify Scientific Research 197
Hugh Pickens DOT Com writes "Michael Hiltzik writes in the LA Times that you'd think the one place you can depend on for verifiable facts is science but a few years ago, scientists at Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology and found only six could be proved valid. 'The thing that should scare people is that so many of these important published studies turn out to be wrong when they're investigated further,' says Michael Eisen who adds that the drive to land a paper in a top journal encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn't a safeguard because the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws. 'The journals want the papers that make the sexiest claims,' Eisen says. 'And scientists believe that the way you succeed is having splashy papers in Science or Nature — it's not bad for them if a paper turns out to be wrong, if it's gotten a lot of attention.' That's why the National Institutes of Health has launched a project to remake its researchers' approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience. 'The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole,' says Hiltzik. 'NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.'"
Re:Scientists == Always Right (Score:5, Insightful)
Science is infallible.
More like science is always wrong. Scientists always set out to be less wrong than the last guy, though.
problems (Score:5, Insightful)
I think the real problem is that scientists aren't lending any prestige to reproducing experiments so nobody bothers. Journals want to publish new results, not confirmation. Advisors discourage students from reproducing experiments, which makes sense since they won't be published.
Peer review isn't about validation (Score:5, Insightful)
Follow-up studies are where the validation/replication/testing happens. This is not new. Any decent scientist knows this. Peer review is a filter, but it's a pretty basic sanity check, not a comprehensive evaluation of the work. Once published, that opens a paper and the ideas within it to critique by ALL readers, not only the reviewers. Thus, post-publication is when the real scientific review happens. Peer review merely removes the stuff that isn't formulated, measured, and organized well enough to bother reading it in the first place (i.e. it gets rejected). It's an imperfect process, so sometimes stuff slips through anyway. That's what the follow-up papers are for.
Replication (Score:2, Insightful)
The best way of checking spurious, biased, or erroneous results is for someone else to independently do the same experiment. However there's no money or glory in replication. So nobody does it.
I wonder which will be most amusing, Fox's interpretation of this story or the tardbaggers' interpretation of that. I've already assigned "herp", "derp" and "6,000 years" to hotkeys.
in vivo biology is not all science (Score:4, Insightful)
Re:Peer review isn't about validation (Score:5, Insightful)
Which is exactly the problem. Nobody's doing follow-up papers. Follow-up papers aren't sexy, don't get published in top-line journals and don't get a lot of cites. In short, they don't advance your career. So scientitsts don't do them.
Re:Scientists == Always Right (Score:5, Insightful)
There's also plenty of data and models out there if you wanted to run your own experiments to confirm or disprove a particular paper or claim. I'd be very interested in reading your counter paper.
Re:problems (Score:3, Insightful)
No, it's not the journals' problem. It's a funding problem.
Nobody wants to pay for scientists to reproduce and verify each others' work.
Science isn't broken. (Score:4, Insightful)
The correct take-away from this kind of study is not that a specific field of science is "broken" (also, cancer research is not all of science), but rather that there is room for improvement.
There is no question whatsoever that cancer research has made leaps and bounds over the last few decades in terms of improving the lives of many people with cancer, both by helping them to live longer, and by helping them to live better. What this kind of study shows is that we can do even better still, if we can find ways to fix the flaws that remain in cancer research.
Re:problems (Score:5, Insightful)
No, it's not the journals' problem. It's a funding problem.
Nobody wants to pay for scientists to reproduce and verify each others' work.
It is both: a funding problem AND the journals' problem. They are not contradictory (far from it, actually).
Re:problems (Score:4, Insightful)
1. How much money they have brought in through grants.
2. How many papers they have published.
3. The prestige of the journals they have published in.
4. How many times their papers have been cited by other researchers.
So, if you keeping your job depends on those 4 things, where is the incentive to check the work of someone else? Especially large, difficult, and expensive experiments. At best, you get a quick "Comment on XYZ" paper that questions some findings and the authors reply with a "Reply to Comment on XYZ" telling you why your comment is rubbish and you didn't understand what they were saying.
Re:Scientists == Always Right (Score:5, Insightful)
He neither said that nor implied it. What he said was that any criticism of AGW is met with a defense akin to a religious fervor. This is a true statement.
As demonstrated.
Re:Scientists == Always Right (Score:5, Insightful)
He neither said that nor implied it. What he said was that any criticism of AGW is met with a defense akin to a religious fervor. This is a true statement.
As demonstrated.
No. Scientific criticism of AGW is fine. But coming up with inane conspiracies, casting aspersions, or character assassinations are NOT valid forms of scientific criticism. Worse, the people often spouting such nonsense have little if any knowledge of the actual science and DON'T WANT TO KNOW IT.
Don't equate denialisim with legitimate skepticism. There are legitimate skeptics, but they aren't the ones claiming that the entire world's population of climate scientists is on a mission to murder Jesus and create a socialist utopia. Deniers make the real skeptics look bad, and actually serve to drown out real scientific skepticism with their idiocy.