How To Better Verify Scientific Research 197
Hugh Pickens DOT Com writes "Michael Hiltzik writes in the LA Times that you'd think the one place you can depend on for verifiable facts is science but a few years ago, scientists at Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology and found only six could be proved valid. 'The thing that should scare people is that so many of these important published studies turn out to be wrong when they're investigated further,' says Michael Eisen who adds that the drive to land a paper in a top journal encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn't a safeguard because the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws. 'The journals want the papers that make the sexiest claims,' Eisen says. 'And scientists believe that the way you succeed is having splashy papers in Science or Nature — it's not bad for them if a paper turns out to be wrong, if it's gotten a lot of attention.' That's why the National Institutes of Health has launched a project to remake its researchers' approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience. 'The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole,' says Hiltzik. 'NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.'"
Re:problems (Score:2, Informative)
This. No one wants to read: "We've confirmed this."
Actually, I might want to read it, and even write it, but good luck getting it into ieee or acm.
Funding for replication (Score:5, Informative)
Re:Scientists == Always Right (Score:4, Informative)
Scientists always set out to be less wrong than the last guy, though.
No they don't. TFA lists many examples of scientists choosing to advance their careers rather than trying to be "less wrong".
The Economist Magazine had a cover story [economist.com] on this issue just last week, that in my opinion covers the issue better than TFA.
Basically, the current system of peer review and replication is failing. Peer reviewers actually miss many errors, rarely check statistics, and almost never re-run any software. The current publishing system has little interest in printing replication, and spending time replicating experiments is a dead end career path. The existing system doesn't work well in the era of "big science" and "big data".
We need to move to a system where all publicly funded science is required to be disclosed when it is initially funded, so negative results cannot later be buried. We should also move to online publishing, with a permanently active area for comments, so if the research is later refuted, or even questioned, that is immediately visible. A portion of public science spending should be set aside for replication. There also should be negative consequences for researchers that publish papers that cannot be replicated, whether because their results are wrong, or because they failed to disclose enough information about how the experiment was conducted. Scientists accepting public funds should be required to make their data and software available.
But the biggest obstacle to reform is researchers and publishers that have prospered under the existing system. Many of them treat the current system of peer review as some sort of holy ritual, and refuse to even admit that the system is broken.
Re:problems (Score:4, Informative)
and getting the grant $$ to repeat a published experiment may be all but impossible.
This is especially true in medicine, where clinical trials can run into millions of dollars.