Meta-Research Debunks Medical Study Findings 261
jenningsthecat writes "From The Atlantic comes the story of John Ioannidis and his team of meta-researchers, who have studied the overall state of medical research and found it dangerously and widely lacking in trustworthiness. Even after filtering out the journalistic frippery and hyperbole, the story is pretty disturbing. Some points made in the article: even the most respected, widely accepted, peer-reviewed medical studies are all-too-often deeply flawed or outright wrong; when an error is brought to light and the conclusions publicly refuted, the erroneous conclusions often persist and are cited as valid for years, or even decades; scientists and researchers themselves regard peer review as providing 'only a minimal assurance of quality'; and these shortcomings apply to medical research across the board, not just to blatantly self-serving pharmaceutical industry studies. The article concludes by saying, 'Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.' I've always been somewhat suspicious of research findings, but before this article I had no idea just how prevalent untrustworthy results were."
For example (Score:3, Insightful)
That fat in your diet is bad for you.
Reality check (Score:5, Insightful)
If medical research were really as close-to-useless as The Fine Summary claims, we'd be hardly better off with modern Western medicine than with homeopathy and prayer. Clearly, we are, refuting the idea that medical research doesn't do a huge amount of good. I'm not saying it isn't flawed, but give it some credit.
Re:Reality check (Score:4, Insightful)
Re:Reality check (Score:3, Insightful)
Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.
TFS seems to be suggesting there is no value in incremental knowledge. No, no you don't discover penicillin every day you go to the lab. But you usually achieve something that advances the state-of-the-art and the things that you learned from your predecessor(s).
Just like democracy.... (Score:3, Insightful)
... it's the most useless way to progress, except for all others.
Be a skeptic, but don't confuse skepticism with truthiness. Unfortunately, I expect a rise in the use of truthiness over science when people will investigate reality.
I couldn't get past the first paragraph (Score:3, Insightful)
In 2001, rumors were circulating in Greek hospitals that surgery residents, eager to rack up scalpel time, were falsely diagnosing hapless Albanian immigrants with appendicitis. At the University of Ioannina medical school’s teaching hospital, a newly minted doctor named Athina Tatsioni was discussing the rumors with colleagues when a professor who had overheard asked her if she’d like to try to prove whether they were true—he seemed to be almost daring her. She accepted the challenge and, with the professor’s and other colleagues’ help, eventually produced a formal study showing that, for whatever reason, the appendices removed from patients with Albanian names in six Greek hospitals were more than three times as likely to be perfectly healthy as those removed from patients with Greek names.
Okay - so I only bothered getting this far into TFA.
Now - I'm no medical junkie, I didnt' even take Bio in high school, but I have occaisonally watched Scrubs and House and ER and a bunch of other medical dramas from time to time.
One thing that always seems to surface in these TV shows is the patients history, like their religion, nationality, where they work, etc. This leads me to believe that maybe - JUST MAYBE - there is actually some correlation between something in the Albanian culture and society that has an increased chance of appendicitis, and that its entirely possible that this pushes doctors towards diagnosing that when some of the symptoms appear. (Not that this is particularily the best course of action, but what else would you do? Run every test?)
But at the same time I know how incredibly innaccurate a lot of television can be about portraying a subject. However, the IT Crowd has basically mimicked my life, but thats another story for another time. Anyways, so if I'm absolutely and completely wrong, feel free to mod me down - but I just thought I'd interject.
Re:Research or the people? (Score:3, Insightful)
There's clearly room for a great deal of improvement here, but people might need to accept that results of medical studies are never going to be as clear-cut as math papers. I think, and now I'm moving into personal opinion, the most important step-- and this applies to both the media and to scientists who might want to cite or cross-reference something-- is that nothing is ever, ever proved after the first paper. Ever. The first paper on some hypothetical connection really should just be ignored, except for prompting further investigation Like I said above, it needs follow-ups, and experiments with different variables controlled for.
Re:Reality check (Score:5, Insightful)
If medical research were really as close-to-useless as The Fine Summary claims, we'd be hardly better off with modern Western medicine than with homeopathy and prayer.
True.
Top notch research is what makes all the medical breakthroughs, but this is only the top few percent of ALL medical research. IMHO one of the main reasons there are so much bogus papers out there is because of the publish or perish attitude in academia, which requires researchers to have a set number of papers published to be eligible for research funding, tenure, other career advancements. I know from experience (although not in medical research, but natural sciences) that sometimes you have to publish a paper even if you know that the results aren't meaningful, or of value to anyone. Then there are people who publish things that were not subjected to rigorous testing, double checking of data, etc. which can easily turn out to be wrong. Lastly there are the cheats. All I'm trying to say is that it's more of a science policy problem than a problem with the integrity of researchers. If the number of publications has to go up, then their quality will surely decrease. Very few research groups (the ones which have good funding) have the luxury of publishing only every now and then. But when they do it's usually a Science or Nature paper. This problem os quality VS quantity is most serious in China. [nature.com] However, not even journals such as Nature are immune to this. [nature.com]
Re:Reality check (Score:2, Insightful)
Are we? What's the evidence?
Most of the improved health we enjoy has more to do with sanitation and nutrition than medicine -- remember to say thanks to your plumber, your garbage collector, your farmer, and your grocer for that. Of the fraction that medicine makes, most of it is due to a handful of advances like vaccination and effective antibiotics (which we are now losing, partly due to faming practices but partly due to bad medical practices).
For the rest, how much of a contribution does medicine actually make? (Especially when you consider the prevalence of iatrogenic illnesses and injury -- the open heart surgery my father had to undergo to replace his mitral valve was amazing, but might not have been needed were it not for the effects of Fen/Phen.) I don't know, and it's not a question that can be waved away by claiming "clearly, blah blah blah." It's not clear, and that's the point of TFA.
Re:Reality check (Score:3, Insightful)
No, it can't, but it can certainly complement it.
Re:Why believe *this* article? (Score:3, Insightful)
Keep in mind when people talk about funding (Score:2, Insightful)
I've heard a lot of disgusted (perhaps accurate, perhaps not) complaints about how politicians (usually this complaint is aimed at Republicans, as in the book title "The Republican War on Science") have "cut money for basic, fundamental research -- the kind that benefits all of us." It's a slight paraphrase, and you might consider it a straw man, but I don't think these gripes are hard to find.
Meta studies like this are worth thinking about when someone says we need more "basic research" in any given area: it might be sincere, well-intentioned (or who knows, even perfectly accurate, in some platonic universe where "needs" could be accurately, objectively assessed and compared), but it might also mean that well-intentioned money would be spent on stuff that is very sciency, but not very helpful.
timothy
Re:I couldn't get past the first paragraph (Score:3, Insightful)
Logic comprehension:
Whether they were 3 times as likely to have them removed or whether they were 3 times as likely to be healthy is irrelevant: the fact is that they were removed under reasoning of Appendicitus, which means they had to have at least correlated to the symptoms of Appendicitus - when the appendix could have been naturally healthy is the claim they are making.
Essentially, all they are saying is that statistically it could have been something besides Appendicitus but because they were Albanian they favoured that particular diagnosis, hence why I brought up my points about Doctors researching family history.
Re:Evidence Based Medicine Movement (Score:3, Insightful)
What? Doctors have to be convinced to follow evidence-based medicine? What were they practicing before? And why are they against it?
You know, there's never been a randomized control trial of the effectiveness of parachutes versus placebo when jumping out of planes at high altitude. Would you care to volunteer?
Some things are just obvious or can be deduced from retrospective analysis or theoretical modeling. And some times it would be unethical or impractical to conduct a randomized control trial of a treatment (e.g., testing an AIDS drug versus placebo).
The idea that doctors are ALLOWED to prescribe drugs for off-label use horrifies me (it's off-label because there's no proof it works for the off-label condition).
No, there is often tons of proof. Off-label just means that the manufacturer hasn't gone through the (expensive, time-consuming) process of proving its effectiveness to the FDA. If the patent has expired there's often no financial incentive to go through the trouble.