Why Most Published Research Findings Are False

Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

It's called

[assert(0)]

publication bias.

One third = Most

[Anonymous Coward]

If one third of the papers are refuted, and this study concludes that "most" of the research is false, does that mean this study is part of the problem? No, i did not RTFA.

Re:Peer review helps

[phantomfive]

It isn't so much a problem of peer review. Peer review has limitations of course, to thoroughly review an article, one would have to repeat the experiment, which most reviewers (for good reasons) do not do. Peer review is good for what it does, give feedback to the authors of the paper, and as you said, it does have a filtering effect.

The other issue is, a lot of papers aren't really worth much at all. Nature might get their share of interesting articles, but in smaller journals, a lot of research ends up being something like, "I had this idea, and I did a small little experiment to see if it was worth anything. Maybe it is." But of course, with a small little experiment, your chances of being wrong are greater: it's just an entry-point for someone else to maybe continue research in an interesting direction. And I have done peer review, FWIW (and if you trust random guys you meet on the internet).

Re:How universal is this.

[Mr. Slippery]

How about the recent stories which prove that glaciers in the north have been *growing*?

A handful of glaciers are indeed growing. The vast majority are shrinking [skepticalscience.com], and they are shrinking much more than the handful of anomalous ones are growing.

A handful of unusual data points in a complex system does not prove a trend. It's as if you were to argue, "Scientists *say* that cigarette smoking will damage your health. But I know one guy who smoked and lived to a ripe old age. Therefore, these `scientific' findings are clearly the result of some politically-motivated anti-tobacco conspiracy."

Re:Peer review helps

[Amenacier]

Peer review doesn't always help. I've studied papers that have the most detailed, thoroughly tested research but end up relegated to some obscure journal because the people peer reviewing the topic don't agree with it. In one case, the pioneers of the field of siRNA lambasted a study which showed that short RNAs can enhance transcription, as well as negatively regulate it. Because this was so far outside of their model for how siRNAs work, they dismissed the work as nonsense, despite the paper showing five replicates of every experiment, and practically putting their entire work, step by step, into the supplementary materials. Papers get into good journals with far less than what I saw for this one, but peer review condemned it to obscurity. I think peer review works, but only if the people reviewing keep an open mind and don't get piqued if the findings disagree with their own views.

FAITH based???

[mangu]

Global Warming is faith based, it's predictions aren't made in anything resembling a controlled scientific environment and the only way to test it's predictions is to do nothing for twenty years and see if the disasters predicted come to pass.

You seem to have NO idea at all about what you are saying. Global warming is based on very exact scientific studies, where "faith" is needed only in that one believes that there exists a reality around us that follows some self-consistent laws.

The studies that present exactly the effects that Dr. Ioannidis mentions are the opposing view, those that pretend to disprove the existence of global warming. Scientist, all over the world, are very strongly in agreement [wikipedia.org] that global warming is an indisputable fact.

The Global Warming Divide

[alexhmit01]

The issues on global warming that are in dispute are numerous... we keep hearing predictions from models that have not been proven to have any predictive powers, and they keep getting more alarmist, and the increasingly ridiculous claims that every example of bad weather is a function of global warming. The issue is the "hockey stick" part of the forward feedback loop... that's the claim that because events will create forward feedback, we will hit a point in a few years where it isn't preventable, because even if we never emitted another CO2 gas, the forward feedback would be self sustaining.

Most things in the universe have negative feedback... The issue with global warming is we know that the current models show this forward feedback, but we KNOW that the models are incomplete. Are we missing significant variables that would create a feedback loop? It seems reasonable to wonder if something will happen with the higher CO2 levels that will cause a negative affect on global warming.

The consequences of GW are dire, and it's a real concern. But the scientific credibility is NOT enhanced by the political advocates calling for the same policies that their fellow ideologues called for for different reasons before, the celebrities weighing in, or the silly exaggerated movies.

Theory of evolution has been tested and demonstrated in small areas with smaller organisms. Theory of evolution is also a concept more than a specific theory, making it easier to demonstrate pieces... yes, evolutionary biology shows the process of small organisms... not the same thing.

Increased CO2 -> increased heat, that's the easy claim
Increased heat -> increased CO2 and there is no way of stopping it is the stranger claim

Perhaps we'll see a spread of CO2 absorbing plants move out of tropical areas to other zones as temperatures change, who knows, but there are plenty of areas for negative feedback, and only time will tell.

Re:Who needs a study: science != medicine/biology

[eli pabst]

No, you are exactly right. His paper was really only intended for the field of population genetics and genetic epidemiology (his fields), where people have been using the standard p0.05 statistical cutoff as their metric for whether a given analysis is significant. So if you have 20 research groups analyze the same question (like is a mutation in gene X responsible for disease Y), according to that methodology by definition 1 of the 20 researchers will find a statistically significant result simply due to chance alone. This is old news and journals stopped accepting papers that *only* had that statistical analysis about 3-5 years ago. Almost without exception, they now require you to show some kind of biological verification (show mutant protein X actually is defective and has reduced activity) OR you can do a replication in a completely independent sample, which is unlikely (again 1/20) to be significant by chance. Unfortunately people have misinterpreted his paper and are applying his point to other fields like chemistry, astrophysics, or even areas of biology where it doesn't apply.

Yes you can test. But keep weaving your sophistry.

[plasmacutter]

yes you can.

Numerous independent climatology models (we're talking virtually every accredited university and think tank on earth with enough resources) based on hundreds of thousands of years of data from geological, oceanographic, and ice core samples, run through supercomputers millions of times.

The only ones that "disagree" are tied to oil companies.

Dental Studies

[Sharkeys-Day]

I read a bunch of dental papers recently, and discovered something rather disturbing. A good 90% or more of studies for dental procedures do NOT use any control group. They all say, "we did X and got the expected result." There is no checking whether the procedure is better than other procedures or even doing nothing at all.

Something to think about next time someone you know is told they need wisdom teeth extracted or some orthodontic appliance.

Re:How universal is this.

[Anonymous Coward]

Let's make this simple. There's no scientific argument on whether or not global warming exists and is caused by humans. An insignificant number of scientists believe that this is not the case.

This is not supposition. This is true. I received the following information from a doctoral student (probably a full phd now) in limnology. I'm going to do him full credit and quote directly:

I haven't read every single post yet, but I did see a large amount of discussion about whether climate change is happening, and whether it has anthropogenic sources. It is and it does. There isn't a debate on the existence point anymore, and there isn't a debate on the fact that it is anthropogenic 2,3,4,5, there are hundreds. Even better and more exact, Oreskes meticulously went through the literature published between 1993 and 2003 in reputable scientific journals related to climate change and did a meta-analysis on them 6. The *928* papers were divided into six categories: explicit endorsement of the consensus position, evaluation of impacts, mitigation proposals, methods, paleoclimate analysis, and rejection of the consensus position.
75% fell into the first three categories ACCEPTING the consensus view
25% talked about methods or paleoclimate, taking no position on recent climate change
0% contested the current consensus position
The last bullet point is the important one. (The description is only slightly paraphrased from the abstract of the paper)

I know not everyone has access to all of the journals and literature, so I did my best to keep to the freely available literature, which often is the top of the line peer reviewed journals anyways. And as a final plea, please please please please verify your points before setting them on the world, especially on topics like climate change, because there is a metric crapton of BS out there, as is the case with any highly publicized issue that has definite monetary consequences.

--Dep

1 Dixon, R. K. et al. 1994. Carbon Pools and Flux of Global Forest Ecosystems. Science 263(5144): 185 - 190.
2 Quayle, W. C. et al. 2002. Extreme Responses to Climate Change in Antarctic Lakes. Science 295(5555): 645 - 650.
3 Hinzman, L. D. et al. 2005. Evidence and Implications of Recent Climate Change in Northern Alaska and Other Arctic Regions. Climate Change 72(3): 251-298.
4 Barnett, T. P., Pierce, D.W., Schnur, R. 2001. Detection of Anthropogenic Climate Change in the World's Oceans. Science 292(5515): 270-274.
5 Thomas, C.D. et al. 2004. Extinction risk from climate change. Nature 427(6970): 145-148.
6 Oreskes, N. 2004. The Scientific Consensus on Climate Change. Science 306(5702): 1686-1686.

Re:Yes...and no!

[Capsaicin]

[I]f you have 66% of published results being found to be wrong you have a huge problem!

I agree. Just as well that a mere 16% were outright refuted then isn't it? :P With another 16% shown to have weaker effects than originally reported. (Ioannidis P A, 'Contradicted and Initially Stronger Effects ...', JAMA 2005;294:218-228.) Moreover, the study was based on 45 papers, with an intentional selection bias, they were both highly cited and claimed high efficacy. Now such citeria might address Dr Ioannidis' particular research interests, but they are hardly representative of the literature over the 13 year period from which they were selected.