Most Science Studies Tainted by Sloppy Analysis 252
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
Yup. (Score:3, Insightful)
as a phd student (Score:3, Insightful)
Sensationalist... (Score:5, Insightful)
My experience (Score:3, Insightful)
Well use the Scientific Method then (Score:5, Insightful)
And one of the first rules is, "Never take a single study as proof of anything! Wait till the results are replicated before you even think of moving to a conclusion."
The major problem is really poor reporting on science research. The news media routinely blazon some **NEW * Scientific * Discovery!!!**. Then you read the story and somewhere around the 10th paragraph you might see that this is based on only one study - and oftentimes even before peer review.
Every scientists knows this. It's a shame the public doesn't. They wouldn't worry so much.
"Most science..." (Score:5, Insightful)
His work seems to focus on population genetics and epidemiology, which is notorious for having unreproducible claims due to a combination of uncorrected multiple testing, publication bias and statistical incompetence. This "gender and genes" is a perfect example: someone does a study, finds nothing, slices and dices the data until he gets p = 0.04 for females or Asians or smokers and publishes his breakthrough finding. I'd have been surprised if he hadn't found almost all of those to be wrong.
If you look at more in-vitro molecular biology and biochemistry work, I doubt if nearly as high a percentage of it is clearly "wrong", although quite a bit of it is worthless.
Re:Sensationalist... (Score:5, Insightful)
Wait for the meta-study (Score:2, Insightful)
Re:How is this news? (Score:5, Insightful)
No Money in Replicating Results (Score:4, Insightful)
That being said, it's my understanding that most scientists work off of grants, and those grants fund novel research. Replicating results is of obvious importance in validating those results, but doing so seems at odds with the funding mechanisms that are the reality for what I would believe to be most researchers.
Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?
(As rhetorical as that might have sounded, I actually welcome those with first-hand experience to respond to it)
Re:Medical research vs. basic research (Score:3, Insightful)
Out of curiosity:
1) What is the usual failure rate for replication?
2) Do the letters routinely get published?
3) You just do that for work you're following up with experiments, not for everything you cite, right?
It must be so... (Score:3, Insightful)
TFA
Since the criterion is that the claim is published, someone had to find the study new and interesting. Most new ideas are going to be wrong, especially true the more significant it is. After all, how many crackpot theories were postulated between Newtonian and Relativistic physics? On the other hand, most things easily verifiable, etc, are too obvious to me considered new and interesting. Note, while I find this interesting, I did not come up with this idea. Some economists published a similar study over a year ago postulating this as a reason. Of course, it's probably wrong.
Sloppy analysis (Score:3, Insightful)
Partly because Medical Doctors dont know math (Score:1, Insightful)
1) Physicians are mostly trained to be healers (practitioners) not scientists.
2) Medical data is very inaccurate compared to the data in standard science (physics, chemistry, geology, etc)
3) Dealing with inaccurate data requies advanced knowledge of matematical statistics and most medical doctors do not have a basic grasp of this field.
4) Many MAJOR ERRORS in the medical literarure are due to the ignorance of basic principles of statistics.
5) Sure ignorance in the field of statistics is only one cause of poor medical research; however it cannot be ignored.
Comment removed (Score:4, Insightful)
Cargo Cult Science (Score:5, Insightful)
Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists [wikipedia.org]. They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.
It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.
Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.
There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.
Re:Yup. (Score:2, Insightful)
Re:No Money in Replicating Results (Score:2, Insightful)
However it's not as bad as it initially sounds. Although we don't receive direct funding to reproduce results, that kind of thing frequently (but not always) happens anyway. If you're building on someone else's work, you will inevitably reproduce some of their experiments. Another example would be that you measure the same thing as someone else, but using a different technique. This kind of corroboration is sometimes even better than directly reproducing their method, because it shows that you can arrive at the same conclusion from a variety of techniques.
So, truthfully, any of the important (and certainly all of the amazing/surprising) results do end up being reproduced in one way or another. But, we never write grant proposals that are simply "we aim to reproduce the work done by X"
I wouldn't object to a tweak to the grant system that gave more recognition to reproducing results and obtaining null-results. But I don't view it as a huge shortcoming of science as it is currently practiced, since we scientists have enough discretion with how we design our experiments that we can put in various checks when they are required.
Re:How is this news? (Score:3, Insightful)
Personally, I think it's true. Between publish-or-perish, financial conflicts of interest, and various political and social movements trying to influence science, a lot of published science is worthless.
However, there's a difference between believing that and actually showing that. Anybody wanting to fix that needs clear and convincing proof first.
Re:How is this news? (Score:5, Insightful)
Clever.
The fact is, good science is hard work. In fact, it is damn hard work, requiring not only a supremely keen intellect but a very high tolerance for tedium, great attention to detail, and usually a big fat wad of cash. Also, it requires a profound lack of ego (and the ability to cope with failure and keep trying), given that a trememdous amount of effort could (and frequently does) wind up being completely discounted by a peer-review or another study.
The endeavor of scientific research obviously provides us tremendous benefits, and is furthering the evolution of our species at a blindingly fast rate (depending on how you look at it, of course). It is very important, very hard, and very expensive.
There are many, many people who would like to be scientists but really don't have the brain for it (as I stated above, it isn't just intelligence that matters). Unfortunately, a lot of them wind up doing research anyway, and they cause problems. Hopefully there are enough good scientists with enough funding to clean up their mess.
Re:Yup. (Score:3, Insightful)
Re:I came here for the global warming ref (Score:3, Insightful)
Just to be somewhat relevant - while there are negative feedback loops, there are also positive CO2/temperature feedback loops. Albedo comes to mind. Not only that, but a lot of the feedback mechanisms are known. The only question is "how much", not "how". The biggest reason people are concerned is that certain things (ice melt, albedo changes) are happening faster than expected, pointing to parameters that were set too conservatively.
Finally, the "ceteris paribus impact" (Hah!) is perfectly well known. Increase CO2 in an atmospheric gas mix, and infrared absorption goes up. End of story.
a pointless statement (Score:3, Insightful)
There are almost no areas of science we're "done" with. The most recent paper on a subject almost always points out where previous papers have gone wrong. Thus, the previous papers have some mistake such as a miscalculation, poor design or incomplete analysis. If you pick any paper published in a peer reviewed journal this month, there's a very high probability that at some point in the future it will be amended or improved by some other paper.
What Ioannidis *has* shown in his recent reports is that in genetics, not enough people are publishing on the same subjects. There are not enough "other papers" out there to check on the previous ones. The result is that papers which in other fields would be recognized as needing improvement are instead treated as the final word.
Most WSJ Articles Tainted by Sloppy Analysis (Score:4, Insightful)
But Dr. Ioannidis has a very narrow definition of science - he only includes statistical studies that use p 0.05 as a threshold for significance. There are, of course, lots of papers that do not show p-values - the purification of a protein, the determination of a genome sequence, the identification of a new fundamental particle. In many cases, p-values are not provided because they are not considered informative - something that happens when the p-value is much much much less than 0.05 (I like my p-values less than Avagadro's constant. With that p-valuep, I think most of my results are correct.)
And, of course, the WSJ misses all of this. The point of the research paper is that you can do everything right, and still be mislead with marginal p-values (0.05). Not sloppy, just not significant enough. We could, of course, require more stringent values, but then we would miss the genuinely rare, but important results.
As the research article points out, results that are reproducible are, in fact, quite likely to be correct. It is perhaps useful to distinguish between science as a paper and science as a process. Most results that stand up to scientific scrutiny over a period of years (that any one cares enough about to validate), are (probably) correct. In some disciplines, which rely heavily on modest thresholds for statistical significance, many results cannot be confirmed.
Exactly. (Score:2, Insightful)
Exactly. Incidentally, this is also the way that the no-global-warming myth started, too.
Re:Flawed medical studies != all of science (Score:3, Insightful)
I'm not sure what depth of statistical analysis you're talking about, but I've found statistical analysis to be exceptionally difficult when applied to Electrical Engineering, and I know I'm not alone here -- it was by far the most difficult post-graduate class at my school.
I can confidently say that nobody who's graduating with me has a complete grasp of all of the statistical tools we were taught. Enough to get by, yes, but most of the things are extremely counterintuitive and easily misused, and this is by people who are really good at math.
I have to laugh when I hear statistics in the news now, because it's all such propaganda. Any time you hear "has been linked to" in an article you know you're about to wade through a steaming pile of bullshit.
Clarification (Score:2, Insightful)