Why Most Published Research Findings Are False 259
Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."
Comment removed (Score:5, Interesting)
How universal is this. (Score:5, Interesting)
At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming. We already has people claiming that the science is wrong and they are generally mocked and ignored because their works are published in major journals. Well, this story seems to indicate that publishing those claims will give them a larger change of it being incorrect.
Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere. So where does this leave the accuracy of the claims in light of how common it seems that they can be wrong even when published in a respectable scientific journal. I know the IPCC looked at them, but they didn't validate any of the claims, they only looks at whether or not Humans were the cause (that was their charter and they acknowledged this in their reporting).
What About Publish or Perish? (Score:5, Interesting)
I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...
Context (Score:2, Interesting)
Re:How universal is this. (Score:2, Interesting)
He is absolutely right (Score:2, Interesting)
Why is it that a large portion of scientific research today is garbage. Well one very powerful reason, money. I saw this firsthand working at a major university medical center on large scale behavior research projects. The outcomes of our studies directly effected existing and experimental drugs and the drug company representatives were right there alongside the researchers at all levels of the process. Professors received "gifts" and other unofficial incentives from them regularly. I saw at least one study where the results were out and out fabricated so that the results would support the effectiveness of a particular drug for treating a childhood psychiatric disorder. In other cases data was included after the fact or blanks were filled in by clinicians from memory. All practices that are highly unscientific. Many of these studies resulted in drugs and treatments for children that are in use today and based on research that is at best questionable and at worst fraudulent. When there is a profit motive behind science it becomes very difficult for it to remain true science and sadly that is the state of affairs in many fields today.
Re:How universal is this. (Score:5, Interesting)
But the concept isn't unique to the pharmaceutical studies. With the general attitude towards dissenters of the Faith that has grew from global warming, I don't see why it isn't true here either. I mean the IPCC used faulty temperature data in their evaluations, Al Gore exaggerated quite a bit and used outdated charts because they proved his point better and Hansen, the guy who pretty much brought Global warming into the lime light admitted to exaggerating claims and justifying it by claiming it was necessary to make people aware of the problems.
I mean it is probably even more prevalent when the data sets used in studies aren't availible to people, the temp data that was proven to be wrong was reverse engineered because Hansen refused to disclose the data. People wanting to review these studies have been mocked and denied access to the data or had the data set obfuscated to make it even more difficult to work with. There was even one instance where someone was told that he couldn't have the data because he was going to pick the work apart and the author didn't want to help him do that. Not very scientific if you ask me. There is definitely room to question what is being said. Most people in disagreement today are in contention over the causes and the purposed solutions which to date, doesn't seem to be helping out in Europe.
Re:Who needs a study: science != medicine/biology (Score:5, Interesting)
oh yeah?
What about Cold Fusion? What about the fabricated nanotechnology data of Schon? What about the memory of water?
The issue is the same with physics, chemistry, and all the others. A large part of the problem is that the top journals *want* papers that can make the news on Thursday; and will select papers that may have not been fully vetted, and also have a bias towards "big shots" (who have much easier time publishing any kind of trash than do young researchers).
Exceptionally rare outliers that were discovered very quickly, and these examples don't jive with the type of problem described in the article, which the GP nails when he points out how it is very concentrated in medical science.
Also, top journals don't "want" papers in the sense that they get the ones they want. Peer reviewers decide what's worth publishing, and I have yet to meet one who feels that an article should be published because it will make the evening news. Big shots do get a big advantage, but in most cases it's because they have a history of good research. Things DO slip through the cracks, but in Chemistry and Physics, those things are within error bars.
The GP's post is so damn good.
Re:Thin Papers Hide Bad Work (Score:3, Interesting)
And you trust that the paper has been seen and deemed correct by a referee. I've been a referee for a couple of Physical Review papers and unfortunately it is indeed rather common that there is too little information in the paper to allow "checking for sign errors" as you call it. So you cannot really trust that the reviewer had enough information to vet the correctness of the paper.
In my case I sent the articles back to the editor with the comment that the authors should first properly explain what they are doing before I can judge the scientific conclusions, together with a long list of ambiguities in the discussion. I'm quite sure though that most referees don't bother since I would say the same about most published papers.
Re:Misleading (Score:5, Interesting)
No question that the corporatizing of research can lead to conflicts of interest, but what are the alternatives?
You mention drugs that kill people. Well, that would be all of them - including sugar pills. In fact, if you created two groups of 1000 people and gave half of them purple sugar pills, and the other half green sugar pills, you'd find that one group vs the other would have a statistically significant increase in heart attack rates some percentage of the time. If you consider the green pills placebos then you'd erroneously prove that purple sugar pills are dangerous. Statistics with 95% confidence are wrong one time in 20...
The same applies to efficacy - especially in stuff like antidepressants where we have almost no understanding of the physiology of the brain. A psychiatrist I was chatting with speculated that it is like treating "cough" back in the 1400s - even if you had penicillian back then it wouldn't be effective against "cough" since doctors of the time had no way of evaluating what the cause of "cough" was and consequently what the appropriate treatment was. It would be a complete mystery to them why one person might miracuously recover with an antibiotic and another would not benefit at all.
I'm all for having independantly-funded clinical trials to test the safety/efficacy of pharmaceuticals, but that would cost taxpayers a fortune. Also - how do you decide what drugs are to be tested? Will we see lobbyists briging congressmen to make sure their companies products get tested before their competitors (leading to huge profits for them)? The problem with publicly funded R&D is that it politicizes research. The private R&D system combined with patents at least prioritizes research that will lead to the largest number of people using a drug/device/procedure (even if affordability becomes an issue).
I'm not actually convinced that corporate malfeasance is the reason for the recent string of drug safety problems. I think a few issues are more significant:
1. Existing drugs work moderately well - so it raises the bar for new drugs in terms of efficacy.
2. Clincial trials are becoming more and more effective at detecting side effects.
3. Doctors tend to assume that well-established drugs are safe. So, even a tiny increase in risk with a new drug leads doctors to avoid it (even if there isn't any strong evidence that older drugs are any better).
4. The tort system ensures that doctors are better off undertreating a disease than risking a side effect. When a patient dies of cancer due to less aggressive treatment it is the cancer's fault. When a patient dies from a side-effect it is the doctor's fault or the drug companies. This neglects the risk/reward tradeoffs that all treatment decisions involve.
5. All the "easy" drugs have already been discovered. This leads to increasing costs for drug R&D and less selectivity for drugs that enter trials.
6. The large number of drugs in development creates enormous demand for clinical trial subjects. This leads to doctors inappropriately enrolling patients and massive costs. Doctors are basically paid by the subject so that have incentive to commit fraud, and more demand means higher fees which means more expensive drugs.
I'm not sure what the solutions to these problems are. Industry consolidation would probably help - fewer companies competing mean that clinical trial costs would drop, there would be less rush, and drug prices would rise so companies have less need to cut corners. Government funding of drug development might not hurt (from start to finish) - the patent issues go away if government just picks up the tab (with all the issues of politicized medical research).
There really are no easy answers though. Sure, people offer easy answers to the pharmaceutical problem, but they don't seem any better than the easy answers to crime, world peace, and all those other things taht people oversimplify...
it's good to hedge your bets (Score:3, Interesting)
Ioannidis wrote a previous article, titled "Why Most Published Research Findings Are False." A very provocative title, one that practically begged the scientific community to read it just to accumulate a laundry list of holes in his argument. A good marketing maneuver. It may or may not be true that most published research findings are false, but the article certainly didn't demonstrate it. Within the context of the discourse he intiated, that would have to be viewed as a kind of willful stupidity, or perhaps marketing brilliance. After all, if the same journal received ten equally well argued articles with titles like "why most research is pretty good," they would still of course prefer to publish his. This truism seques nicely into the new article (on which he is not first author), which is titled, "Why Current Publication Practices May Distort Science." Use of the word "may" is quite helpful here. Does the article live up to its title? It may not be convincing, but it would be even less so without the word "may."
Re:The Global Warming Divide (Score:3, Interesting)
They do have predictive powers. They predicted a decade ago the trend in markedly increased storm severity and frequency.
They predicted the droughts and climate changes which are afflicting numerous regions world wide, including france, spain, and my area of the US.
The issue with global warming is we know that the current models show this forward feedback, but we KNOW that the models are incomplete
This statement is intellectually dishonest
we also know the model of the universe (including most of our scientific theory) is incomplete. This doesn't stop us from applying the current model to everything from the production of your computer to nuclear power/weapons to deep space exploration.
Increased heat -> increased CO2 and there is no way of stopping it is the stranger claim
and this is still considered a fringe theory among the vast majority (real scientists rather than those charted by oil companies to "sow controversy") who believe in global warming.
All or nothing is not how science works.