Why Most Published Research Findings Are False

Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

Peer review helps

Peer review no doubt helps to limit people who intentionally want to cause problems. Sokal's bullshit paper on quantum gravity (see The Sokal Hoax [amazon.com] ) made it into print only through a non-peer-reviewed journal. While it is disturbing to think much published scholarship is unreliable, at least it isn't necessarily malicious.

Re:Peer review helps

Peer review can both help and hinder - there's the reputation effect of guest authorship where having a well-known, senior, academic's name on the paper helps it through no matter how absurd the findings.

Then there are reviewers who review papers they do not have the expertise to review. And to be frank I've seen some pretty bloody ludicrous comments from supposedly expert reviewers - the sort of stuff 1st year students wouldn't make.

But I do think that the majority of researchers are dilligent and beleive in what they submit. And lets face it - if it is an emerging area and you have a neat result that either refutes someone else's grand theory or is just really novel you're going to want to see that in print. It is because we seek to replicate research that findings are later falsified. This isn't evidence that the system is broke it is pricesly how it should work. It is the work that can't be falsified that stands the test of time and contributes to our knowledge.

If there are people who think that falsifying published research is somehow a bad thing - that is shows there's a problem in research standards - the they really really need to go back to school and read some Karl Popper.

reviewing papers without expertise

Indeed in my field (a sub-area of computer science) people are usually highly skeptical of any supposedly important new result in the field that was first published in one of the highly prestigious but generalist journals, like Nature or Science. These often end up being, if not outright wrong, at the very least seriously over-extending their claims or the importance of their claims, in a way that would never get them published in a specialized journal filled with an editorial board who were actually experts in the specific area in question.

This is only exacerbated by the fact that, because generalist publications know they don't have expertise in every specialized area on staff, they often ask the authors to suggest potential reviewers of their own papers. Of course, authors are likely to suggest reviewers who they think will like the paper, not the ones who would give it a grilling.

I think the interest of this particular study is not so much that a lot of science turns out to be wrong, but that a lot of the most prestigious publication venues turned out to be wrong more often.

Obvious question ...

How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?

Re:Obvious question ...

Actually, someone already has. [slashdot.org]

Who needs a study: science != medicine/biology

How long until some researcher releases a study showing that Dr. Ioannidis' research findings are themselves wrong?

Who needs a study? Simply reading the article shows that he has fallen precisely into the trap that he is complaining about i.e. overstating his results. He forgets one very simple point: not all science is medicine/biology.

As a particle physicist I would strongly disagree with his conclusions, at least as applied to experimental particle physics. It is certainly true that some papers turn out to be wrong but this is rare and usually ends up as a 'big thing' in the field. Outside my field I'd be very surprised if the majority of physics or even chemistry papers turn out to be wrong (but I certainly not a chemist so this is just my impression).

As for medicine I can certainly see that they have a problem. Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.

So, Dr. Ioannidis either show us some data from chemistry, maths and physics or stop complaining that all of science has a problem on this scale. From where I stand your evidence points to a problem with bioscience/medical research only.

Re:Who needs a study: science != medicine/biology

I would argue that this problem is not only pretty much non-existent in chemistry and physics, but that even biology, at least cell and molecular biology do not have this issue either. Typically when a biologist publishes a protein structure or sequences an organism's DNA no one shows up later and says it is wrong. In fact, it's rather large news when it does.

For example, there was a bit of a controversy over protein crystallographers recently. A person had published a paper on a protein structure that seemed to contradict all previous though functions for the protein. It turned out that they had used the wrong parameters in their phasing program. However, this doesn't happen in most to most papers, and certainly not a majority of them.

I would say that this problem is mostly specific to medical research. By its very nature, medical research is a good deal more prone to human fallibility since both subjects and researchers are human beings.

Science is supposed to work like this.

Afterall how many times have we been told "don't eat X/do Y it is bad for you" only later to find out that actually it isn't half as bad as they thought and may even have benefits? Just because a lot of medical research is often flawed does not mean that all of science has the problem on the same scale.

The problem here is that the popular press always report the very latest 'finding' in what is a complex field. Yet we should know that not only in medicine, but in virtually all experimental sciences, a single paper is not sufficient to establish some new profound truth.

Dr Ioannidis' largest problem is that he thinks he has identified a problem. There isn't one. This is how science is supposed to work! We publish methodologies so that the work can be replicated by other teams. Some findings survive futher scrutiny, some don't. The "hotter" the field, the less you are going to rely on the latest single study, no?

So he's found 1/3 of studies were refuted, but later work. Great, they were refuted, what's the problem? And how do we move from that to the conclusion that "most" scientific papers (even outside the hotter fields of bio-medical research) are wrong. And what about looking at outcomes? The advances of medicine even in my lifetime are astounding, this is hardly the result of a system that isn't working!

Re:Who needs a study: science != medicine/biology

oh yeah?

What about Cold Fusion? What about the fabricated nanotechnology data of Schon? What about the memory of water?

The issue is the same with physics, chemistry, and all the others. A large part of the problem is that the top journals *want* papers that can make the news on Thursday; and will select papers that may have not been fully vetted, and also have a bias towards "big shots" (who have much easier time publishing any kind of trash than do young researchers).

Exceptionally rare outliers that were discovered very quickly, and these examples don't jive with the type of problem described in the article, which the GP nails when he points out how it is very concentrated in medical science.

Also, top journals don't "want" papers in the sense that they get the ones they want. Peer reviewers decide what's worth publishing, and I have yet to meet one who feels that an article should be published because it will make the evening news. Big shots do get a big advantage, but in most cases it's because they have a history of good research. Things DO slip through the cracks, but in Chemistry and Physics, those things are within error bars.

The GP's post is so damn good.

Re:Who needs a study: science != medicine/biology

Definitely not most. All. The process of science is using theory to predict a result, carrying out an experiment to test whether that result occurs or not, and revising the theory if necessary.

We cannot ever prove that the current theory is, in fact, "correct." For all we know, there is some rule encoded into the stuff of reality that gravitation will reverse itself next Tuesday, and we can neither disprove this nor predict it. All science can offer is the minimally-complex theory that fits all currently known data.

Re:Obvious question ...

First we find that 89.2% of all statistics are made up

...and the other 11.8% have arithmetic errors.

Misleading

Title is wrong. It says that the FDA is corrupt. And that published papers take around 3years to get peer reviewed where the bad ones are removed. What a blatent attack on science generally. Sure paper publishing needs to be reviewed but 'most published research is false' is an outright LIE. 'Most published research' includes all of our basis of scientific knowledge. If most of our theories on biology were wrong really we realistically wouldnt have been able to move forwards into working with genes if we didnt know what a cell did.

Re:Misleading

No question that the corporatizing of research can lead to conflicts of interest, but what are the alternatives?

You mention drugs that kill people. Well, that would be all of them - including sugar pills. In fact, if you created two groups of 1000 people and gave half of them purple sugar pills, and the other half green sugar pills, you'd find that one group vs the other would have a statistically significant increase in heart attack rates some percentage of the time. If you consider the green pills placebos then you'd erroneously prove that purple sugar pills are dangerous. Statistics with 95% confidence are wrong one time in 20...

The same applies to efficacy - especially in stuff like antidepressants where we have almost no understanding of the physiology of the brain. A psychiatrist I was chatting with speculated that it is like treating "cough" back in the 1400s - even if you had penicillian back then it wouldn't be effective against "cough" since doctors of the time had no way of evaluating what the cause of "cough" was and consequently what the appropriate treatment was. It would be a complete mystery to them why one person might miracuously recover with an antibiotic and another would not benefit at all.

I'm all for having independantly-funded clinical trials to test the safety/efficacy of pharmaceuticals, but that would cost taxpayers a fortune. Also - how do you decide what drugs are to be tested? Will we see lobbyists briging congressmen to make sure their companies products get tested before their competitors (leading to huge profits for them)? The problem with publicly funded R&D is that it politicizes research. The private R&D system combined with patents at least prioritizes research that will lead to the largest number of people using a drug/device/procedure (even if affordability becomes an issue).

I'm not actually convinced that corporate malfeasance is the reason for the recent string of drug safety problems. I think a few issues are more significant:

1. Existing drugs work moderately well - so it raises the bar for new drugs in terms of efficacy.
2. Clincial trials are becoming more and more effective at detecting side effects.
3. Doctors tend to assume that well-established drugs are safe. So, even a tiny increase in risk with a new drug leads doctors to avoid it (even if there isn't any strong evidence that older drugs are any better).
4. The tort system ensures that doctors are better off undertreating a disease than risking a side effect. When a patient dies of cancer due to less aggressive treatment it is the cancer's fault. When a patient dies from a side-effect it is the doctor's fault or the drug companies. This neglects the risk/reward tradeoffs that all treatment decisions involve.
5. All the "easy" drugs have already been discovered. This leads to increasing costs for drug R&D and less selectivity for drugs that enter trials.
6. The large number of drugs in development creates enormous demand for clinical trial subjects. This leads to doctors inappropriately enrolling patients and massive costs. Doctors are basically paid by the subject so that have incentive to commit fraud, and more demand means higher fees which means more expensive drugs.

I'm not sure what the solutions to these problems are. Industry consolidation would probably help - fewer companies competing mean that clinical trial costs would drop, there would be less rush, and drug prices would rise so companies have less need to cut corners. Government funding of drug development might not hurt (from start to finish) - the patent issues go away if government just picks up the tab (with all the issues of politicized medical research).

There really are no easy answers though. Sure, people offer easy answers to the pharmaceutical problem, but they don't seem any better than the easy answers to crime, world peace, and all those other things taht people oversimplify...

How universal is this.

At the risk of being modded down to oblivion, I am still curious to how this effects popular theories like global warming. We already has people claiming that the science is wrong and they are generally mocked and ignored because their works are published in major journals. Well, this story seems to indicate that publishing those claims will give them a larger change of it being incorrect.

Anyways, it seems that if you don't tow the line on climate change, there is no room for you anywhere. So where does this leave the accuracy of the claims in light of how common it seems that they can be wrong even when published in a respectable scientific journal. I know the IPCC looked at them, but they didn't validate any of the claims, they only looks at whether or not Humans were the cause (that was their charter and they acknowledged this in their reporting).

Re:How universal is this.

The idea behind this is that pharmaceutical studies are difficult and expensive to perform, so they rarely get challenged for some time, and when they do, the challengers are more often in more obscure journals: so when people go to cite statistics and findings, they don't notice the fact that what they're quoting has been invalidated. Climate change has been studied over and over again and subjected to extensive analysis by many minds for many years, particularly because so many people have questioned it. To suggest what you are saying now is a little behind the times.

Re:How universal is this.

But the concept isn't unique to the pharmaceutical studies. With the general attitude towards dissenters of the Faith that has grew from global warming, I don't see why it isn't true here either. I mean the IPCC used faulty temperature data in their evaluations, Al Gore exaggerated quite a bit and used outdated charts because they proved his point better and Hansen, the guy who pretty much brought Global warming into the lime light admitted to exaggerating claims and justifying it by claiming it was necessary to make people aware of the problems.

I mean it is probably even more prevalent when the data sets used in studies aren't availible to people, the temp data that was proven to be wrong was reverse engineered because Hansen refused to disclose the data. People wanting to review these studies have been mocked and denied access to the data or had the data set obfuscated to make it even more difficult to work with. There was even one instance where someone was told that he couldn't have the data because he was going to pick the work apart and the author didn't want to help him do that. Not very scientific if you ask me. There is definitely room to question what is being said. Most people in disagreement today are in contention over the causes and the purposed solutions which to date, doesn't seem to be helping out in Europe.

Re:How universal is this.

> I fail to see how you can draw any conclusions about the reliability of atmospheric physics papers from a study of biomedical research papers.

Biomedical research is a lot more amendable to verification and falsification, thus an argument can be made that errors are getting corrected. Global Warming is faith based, it's predictions aren't made in anything resembling a controlled scientific environment and the only way to test it's predictions is to do nothing for twenty years and see if the disasters predicted come to pass. Now consider that rerunning a medical test and the origional paper wrong will get a researcher rewarded while writing anything whatsoever questioning human caused global warming gets a researcher labeled a whore of the oil companies and the argument that the science on GW might be at least as flawed as these biomedical papers grows.

Re:He's merely observing the obvious, and no.

In order to test the Global Warming theory you need 2 carbon copies of 1900 earth

That wouldn't work, as the copies would be comprised mostly of carbon powder, so the geophysics would be completely different.

What About Publish or Perish?

I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...

Re:What About Publish or Perish?

How true that is.

The significant other is quitting grad school as soon as she gets her Master's in Neruoscience(she's in the PhD/Master Program). She can't stand the constant pressure of publishing nor the need constantly justify grant writing. She's not the best researcher, but the pressure is enough to drive her to not caring anymore. She'll get her consolation prize and get on with her life.

Maybe she's just not cut out for academia, though it's losing out on the great potential she has.

Re:What About Publish or Perish?

I would think that "Publish or Perish" must contribute to a lot of crappy papers getting published. Shovel it out the door, somebody else says it's wrong, write another grant for a study to verify that, shovel that one out the door, rinse, lather, repeat...

It does indeed. Thirty years ago an assistant professor could get tenure by publishing one good paper per year in an archival journal. Nowadays an assistant professor is expected to publish four or more journal papers per year. This leads to the well-known academic concept of the "MPU", i.e. the minimum publishable unit, or "just how many papers can I squeeze out of this one good idea?". This also leads to the backwards situation where a senior professor sitting on a Promotion & Tenure Committee may have fewer published papers (and fewer awarded research dollars) over his entire career than the assistant professor whose tenure he is voting on. Believe me when I say that the hypocrisy of this double standard is not lost on the junior faculty.

There's no doubt in my mind that the signal-to-noise ratio in archival journal papers has plummeted in the past two decades. 90% of all journal papers are superfluous, repetitive, or lacking in any significant advancement of the art, and I'll plainly admit that includes my own papers. Everyone in academia realizes what's going on, and knows it isn't good for the students or the faculty, but unfortunately that's the way the beans get counted in the academic world.

We apologise again for the fault in the research

Those responsible for refuting the research of the people who have just been refuted, have been refuted.

Follow the incentives

Basic idea: high-profile journals want papers that are new and exciting. This means that scientists have an incentive to 1) rush their work, 2) choose fields that are popular, and 3) claim that their papers solve more than they actually do. This leads to sloppy, dishonest papers.

I'm not going to judge this paper - I haven't read it thoroughly - but to pair a title like "Why most published research findings are false" to a pretty well-known problem seems itself like an example of problem 3!

And this is bad why?

Scientific research is just that -- research. If it were as easy as doing a couple of experiments, revealing the "truth" and moving on to the next thing, we'd all be living around Alpha Centauri by now. But science is hard and therefore a lot of conclusions are naturally going to be wrong. If that weren't the case then we wouldn't even need any scientific journals -- all we'd need would be newspapers.

Remember the whole "theory of evolution" issue that the creationists keep harping on? "They call it a theory so it must not really be true?" We all know that evolution is just about as "true" as any science gets -- and yet surely there are some portions of the current body of knowledge about evolution that will one day be falsified by later research. That's not a bad thing.

Notable research that has since been thought to be flawed or insufficient: Newtonian physics. Niels Bohr's model of the atom. Gregor Mendel's research into genetics. Einstein's theory of general relativity. Koch's postulates for determining disease causation. Quantum mechanics. And so on.

Unclear what they mean

Even after reading the article, I'm still not sure if the authors are saying:

A) Given that research has been published, it is more likely to be false than not; or
B) Given that research is false, it is more likely to be published than is the case for true research.

I mean, it says:

Dr Ioannidis made a splash three years ago by arguing, quite convincingly, that most published scientific research is wrong.

So, (Wrong Articles)/(Total Articles) = >=0.5, right?
But the only figures I can find in the same article are:

Dr Ioannidis based his earlier argument ... on a study of 49 papers ... (H)e found that, within only a few years, almost a third of the papers had been refuted by other studies.

So.. "most" is now "less than one third"?

I'm somewhat alarmed that The Economist lets people who don't seem to grasp basic statistics write their articles.