Meta-Research Debunks Medical Study Findings 261
jenningsthecat writes "From The Atlantic comes the story of John Ioannidis and his team of meta-researchers, who have studied the overall state of medical research and found it dangerously and widely lacking in trustworthiness. Even after filtering out the journalistic frippery and hyperbole, the story is pretty disturbing. Some points made in the article: even the most respected, widely accepted, peer-reviewed medical studies are all-too-often deeply flawed or outright wrong; when an error is brought to light and the conclusions publicly refuted, the erroneous conclusions often persist and are cited as valid for years, or even decades; scientists and researchers themselves regard peer review as providing 'only a minimal assurance of quality'; and these shortcomings apply to medical research across the board, not just to blatantly self-serving pharmaceutical industry studies. The article concludes by saying, 'Science is a noble endeavor, but it's also a low-yield endeavor ... I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life.' I've always been somewhat suspicious of research findings, but before this article I had no idea just how prevalent untrustworthy results were."
Quack Attack (Score:5, Interesting)
The problem I know because I am (retired) RN. I went to my MD for a severe sinus infection and chest infection some years ago. During the visit I asked for Abuterol Inhailer to assist with clearing my chest. This is standard Respiratory Therapy stuff. They gave me a puffer which I took home, used 10 or 15 puffs out of and threw it in the drawer. Ever after that BCBC has me as Diagonsis Asthma. I am not asthmatic. This will screw up my healthcare for the rest of my life! Makeing all of these errored stupid databases cross link will do far worse than this. My daughter (age 23) was emergency taken to the local hospital with what appeared to be an Epileptic Seizure. Consequences included she couldn't drive for 6 months! I will skip the details, her seizure was a cardiac seizure. It took her actually taking her case to the local Fire Department to get a heart monitor strip to make this undeniable. She is now treated well but forever she will be DX Epileptic even though it is completely wrong. Does any sane person want this sort of a system where you cannot go to another doctor and have him/her look at you rather than some record first? Who wants in that trap? If you are an MD in that trap even if you see that the record is wrong, you can go to jail, lose your job etc all if you go against this insane record that is completely in error. Please wake up people this is a prison without walls! You cannot escape! You will have to leave the country to get away from a bad diagnosis or a stupid keystroke error. Remember the computers have a forever memory and no intelligence.
Re:Reality check (Score:5, Interesting)
I skimmed TFA, and it's important to note that scientific base research (for new pills, procedures, etc.) is not the issue here. This is about studies, i.e. field testing of large numbers of patients, and the (wrong, causation != correlation, etc.) interpretations that are made public afterwards. Funny enough, until recently, criticising the official results of medical studies was seen as conspiracy theory by those in power in medical circles.
Re:Quack Attack (Score:3, Interesting)
Anytime a record follows a person there needs to be some form of error checking involved and the ability to fix it.
Re:Reality check (Score:3, Interesting)
Yeah, but what if they are being told over and over again by "alternative practicioners" that the medicine doesn't work, and has terrible side effects? That has a placebo effect, too.
Re:Reality check (Score:2, Interesting)
I think the implication is that "low-yield" generally means "unprofitable", so the honestly involved in the claimed results a scientific endeavor has a lot to do with how much profit is expected from said results.
Note that TFS says "major improvements", not "improvements". Pharma/Med companies generally only profit significantly from "major improvements".
If research were to uncover a drug that is 1% more effective in preventing hayfever than existing prescription meds, that would be an "improvement". Hardly a "major" one, but an improvement. But it costs a lot of money to put a drug to market.
So I, as the company that employs that scientist, have five choices:
1. Fund a completely honest study and safety testing at a cost of $X so the product can go to market and the consumer will say "1% improvement? Ho, hum, I'm happy with what I have." I lost money, but maybe a few patients ended up with a better quality of life.
2. Decide it's unprofitable, but publish the results so anyone can freely use it, since it's not worth the effort for me to pursue it. Maybe someone else can use it as a foundation to improving the state of medicine. I won't profit from it, but the world might be a better place.
3. Continue funding the project in the hopes that further advances become more profitable. Maybe I'll make money someday, maybe I won't. Ya rolls the dice, ya takes yer chances.
4. Tell the scientist to stop wasting time on it, assign them to another project, and have their notes shredded so at least competitors can't use it for something.
5. Fund a study that demonstrates a completely revolutionary breakthrough in effectiveness, advertise it as such, push it for a lot of off-label uses, send some doctors some nice pens, hope the placebo effect holds when people switch to it, and milk that mooing cash cow as long as my patent holds out, then shove it to OTC and come out with a very-slightly-tweaked new version of it immediately thereafter that has a couple of random letters after the name in the hopes I can get people to think it's a different med, and repeat the cycle as many times as there are two-letter combinations.
I've ranked them in ascending order of likelihood.
These are well known problems (Score:5, Interesting)
I don't want to discount the value of the study itself. Clearly it's important to quantify how bad the problem is and try to develop solutions. But at the same time, the article and summary might give one the impression that the errors and biases involved were newly discovered by the researchers. A few examples:
The secondary marker problem (e.g. tracking cholesterol levels instead of real outcomes like deaths)
Comparing new drugs only to placebo or only to drugs that aren't best-in-class or using an intentionally weak dose of the comparison drug
Using meta-analysis of other studies instead of doing new research (and often doing it badly)
Doing retrospective analyses like chart reviews instead of prospective studies (and often doing those badly)
To expand upon that last example: common problems with the methodology of chart review studies were investigated thoroughly by Gilbert and Lowenstein in 1996 [nih.gov]. Despite their findings and recommendations for how to do a chart review properly, things haven't improved much since.
Many doctors and researchers have been critiquing studies and warning about these problems for years. In the emergency medicine context, for example, Jerome Hoffman, a UCLA medical professor and emergency department physician, is well known as a critic of poorly designed studies in the emergency medicine literature. He has critically reviewed studies since 1977 as part of a continuing medical education program called Emergency Medical Abstracts [ccme.org].
So the problems are well known. The bigger issue is how to fix them.
Jeez. (Score:1, Interesting)
a meta study? right there is suspect.
Now there are things you need to understand when reading a study.
A)Sometimes the conclusions do not match the data. This rarely happens in proper qualified journals, but it does happen.
B) Understand what p value means and when it's significant.
C) Understand blinding.
D) Understand sample size.
E) people make mistakes, but that doesn't necessarily mean a study is wrong. In fact, very few times is any one flag mean the study is bad.
F) many things called studies, aren't really studies. Of course, he lumps anything the uses the word 'studies' into the same category. Effectively saying TV is just a valid of a place to get studies as is NEMJ.
t seems to me, that this guy like to take certain situation and that extrapolate them to ridiculous levels. I wouldn't trust him to give me directions. He cherry picks data and then applies the sharp shooter fallacy.
He also doesn't seem to know there is a difference between blinded, double blinded and triple blinded.
Re:For example (Score:1, Interesting)
I've been on LCHF for a while and lost 21 lb in nine months. I haven't worked out, I sit in front of computers day and night.
First months I gorged myself in as much fried pork as I could stand, with delicious lard and all. Lost 5 punds.
My blood pressure has dropped to high-school levels. Going to get a blood analysis next month, expecting improved overall.
Meta-researchers (Score:3, Interesting)
Why I don't doubt that some good critical thinking, and legitimate questioning come from these meta-research studies, I fear that the process is ripe for abuse, as basically being so awash in data (information overload) that given enough data you can pick and choose to fit your a priori or posteriori hypothesis.
I applaud the increased scrutiny of statistical analysis, which is truly difficult to administer on even the best designed and controlled biological and medical studies, where you have very little "total control" of the experimental subjects - damn ethics committees on testing human subjects, and using double blind testing is the best you can do to eliminate bias, yet may mask discovery of experimental flaws during the testing phase. Things go "wonky" in strange ways, for example testing a heart medication, and a freak snowstorm skews all the results because of the rise of heart attacks from the increase in shovelling. We can't legally put 1000s of humans in vats for 10-20 years to test everything, and computational models are primitive and only address what the model is designed to look for, while most medical testing focuses on the unexpected results and effects that may only appear in a small fraction of society, yet if the consequences are dire, it can kill an entire potentially life-saving product.
I fear that the "undergrad social science" approach of meta-study research will make the approach stained with a reputation of people who want to "do science", but without the messy get-your-hands-dirty that costs money (an increasingly mythical subatomic particle in most fields of science and labs around the world) and just do a PR-style re-spin (think: re-branding) of the results of multiple similar but different experiments to reach a conclusion that was not considered by the original experimenters, so whom may not include appropriate experimental controls to minimize draw incorrect conclusions from this re-interpretation of the experimental data. Of if they are really lazy (like social science elective takers), draw conclusions from a compilation of results, and not even bother looking at the original (raw or filtered) data at all. I guess I'm trying to say that there is limited latitude for re-interpreting data for anything beyond what the experiment was designed to test. It can be very useful for detecting and thwarting bad or biased experiments, but as far as I know, it cannot produce trustworthy results from bad experiments.
Re:For example (Score:3, Interesting)
Neither fat nor carbs are bad for you. It's the extreme exclusion of one or the other that is bad. You know vegetables, fruit and such are carbs, right?
Extremes work in the short term but long term your health will suffer.
This is the nutritionist line that comes up every time someone mentions a low card diet.
There are all the dread long term affects, never very specific, and always based on the assumption someone will stop eating carbs for the rest of their life, or never eat any carbs at all while on a low carb diet.
This just a perpetuation of the misunderstanding of the low carb diet. Yet its been proven in the Military, proven even by organizations that were vocal foes for years [cbsnews.com]. Every serious study [scientificamerican.com] has supported the low-card diet.
Low card does not equal No-Carb. And even very-lo-card start-out plans (Atkins) doesn't mean forever. There are simply no studies to support the scare mongering about lo-carb diets.
Re:For example (Score:2, Interesting)
Then you needed to eat fewer calories, and not just from carbs. Low carb diets work only when caloric intake decreases. (If you consume 3000 calories of fat and protein and burn 2000 calories, just what do you think happens to that other 1000 calories?)
In fact, people can lose weight on either low carb (preferable a vegetarian low carb [unreasonable.org], if one doesn't want to shorten one's lifespan, since a typical low carb/high protein diet has detrimental effects of coronary blood flow [nih.gov]) or high carb [ama-assn.org] diets. The problem is caloric intake, not the proportion of macronutrients in the diet. If carbs are to blame, why does Japan have one of the lowest obesity rates in the world [globalpost.com] and a diet still centered around rice? And why is that obesity rate increasing as the diet Westernizes and becomes less carb-centered? It's nothing to do with carbs versus protein or fats, it's serving size, sugar, and exercise patterns.
Yes, and the belief in the effectiveness of low carb diets is just evidence of this.
Anyway, congratulations on decreasing your caloric intake and losing weight, even if it took belief in the effectiveness of pseudoscience to help you do it.
Re:For example (Score:3, Interesting)
The only thing that reliably results in weight loss is lower intake of calories. Low-carb diets do this because most of our intake of calories is via carbs. It's not rocket science, and it's not easy, a lot of the time. "Low carb" diets are actually just "eat less" diets....
Re:newsflash! or why no followup study (Score:2, Interesting)
It is rare for any medical journal to publish negative results, and yet that is precisely what we need to advance science.
Both positive results - and the failures that prove they were flawed. But journals only publish the former, not the latter.
Re:For example (Score:1, Interesting)
How it is explained when you want to loose weight.
You exercise for strength. You eat to loose weight. I lost 30 pounds by doing the same as the gp poster. Many people have. They even have a name for it 'the adkins diet'. It doesnt mean 0 carbs. It means less than what you need for awhile (the weight loss), then the balance (eating only what you need to maintain). As you age your body changes too, so you will have to revisit quantity every few years. So what you ate as an 18 yr old no longer holds true when you are 50. After I lost the 30 I went back to a similar diet I had before but in smaller quantities.
If you do *ENOUGH* exercise you can loose weight. It however takes a serious amount more than most people realize (think 3-5 hours of strenuous exercise per day every other day). But it really is about balance and finding what your body really needs.
Me I need some where between 30-80 carbs per day given the way I am at work. I added it all up and I was doing nearly 300 a day. So yeah I fattened up as I aged. Slimmed it to about 10-20 per day and I lost the weight and now eat just enough so I dont gain or loose (about 30-60).
Fat != no strength either. I had an old tv weighed about 250 pounds. I couldnt push the thing. This one big dude I knew could pick the thing up. EASY... He is what most people would call obese. Yet he is by far stronger than most people I know.
Now if you are looking for tone. You need to strictly control carbs and calories. As you will need calories to burn while you exercise or you will feel very weak. Also do not exercise every day. But you need to give your body time to heal after every time you basically beat it up. I used to do 3 days a week M,W,F. At first I was doing every day and getting no where. When I switched to 3 days a week I toned up very nicely. Had to stop though as I overexerted my knees :( Joint problems are also an issue so be careful. As these do not strengthen up as fast as your muscles will and you can literally tear them up. Part of the reason you must have carbs. As fat is what keeps your joints working. If you are exercising you actually want a smidge more carbs than I do as you will also burning them up. Your body still needs these things. Also be ready for some killer headaches as your bloodpressure freaks the hell out.
Going on a diet is a fad. Changing the way you eat is a lifestyle change.
Good rule of thumb for food though is size of your fist if you are not exercising. Size of both your fists put together if you are. If you want to accelerate things half that and eat 6-8 times a day. Then do two smaller sessions of exercise on your exercise days.
Re:Reality check (Score:3, Interesting)
Certainly the Slashdot Summary is misleading and hyped, but that's SOP. However, the article still is a bit more damning. Not only does 'inflating' the results of studies help the academic publishing / funding treadmill, but the same mechanism inflates the 'value' of medications and medical interventions.
That is a huge problem, especially in the US where the trend for Medicare / Medicaid (the country's largest insurers) is to mindlessly pay for 'new and better' as well as the general desire for new and shiny things that go 'ping'. New drugs that are really not much better than old ones (but much more profitable). New procedures that aren't much better than old ones (but much more profitable), new machines that aren't much better than old ones (but now that you've paid off the last one, can we sell you version 2 with tint control?). So this sort of sloppy science isn't just a problem in at the chalkboard. It's a problem in the wallet and in fact, a problem that risks people's lives.
Just one example: Bone marrow transplant [amazon.com] for Breast Cancer. Heavily touted. Heavily advertised. Lawsuits again insurers that wouldn't pay for it even though much of the early research was clearly substandard and the benefits marginal. But it hit people at their heartstrings - dying women - and there was a huge push for it. Later research showed no benefit for a procedure that really put patients through an awful several months (worse than the cancer).
It has become a perfect storm in which the complexity of human biology, the poor training of medical researchers (MD / PhD programs basically create physicians with weak clinical skills and researchers with weak research skills), the enormous financial and academic stakes of the research and everyone's interest in 'getting better' or simply staying alive have created (at least in the US and I suspect to a certain degree in other first world countries) an expensive and poorly controlled industry that yields marginal patient benefits and incurs enormous costs - both financial, social, moral and physical.
Re:For example (Score:5, Interesting)
For a year and a half I tracked every ounce of food I ate, recorded it in software I wrote, meticulously recorded calories burned on exercise machines (this is the largest source of error, but I did my best to compensate) and found the following:
Graphing my actual weight and my projected weight using (start weight - ((2660 - calories eaten + calories exercised) / 3500) will never be more than 3 pounds apart, even over a weight change of over 100 lbs, regardless of whether my caloric intake was from a tub of Crisco or from cucumbers dipped in virgin blood. Conclusion: eat whatever the hell kind of food you want, just keep the calories low, your long-term weight change will be the same. But it's easier to eat fewer calories, in my experience, if you cut out carbs. They make me hungrier.
The +/- 3 pounds fluctuates based on when you last peed, how much you drank and when, and how much meat is sitting in your colon pending expulsion.
I see your problem right there: "Science" (Score:5, Interesting)
Doctors are notoriously bad at doing scientific investigations, and since peer review journals are reviewed by peers, they don't know bad science when they see it.
I've been arguing with my father about this for... what, fifteen years?!? He subscribes to the New England Journal of Medicine and is forever telling me about the latest study telling you to that standing on one leg reduces breast cancer in nuns. These studies are, without fail, trivial to eviscerate as their control groups are either non-existent or very poorly picked.
Doctors just don't know much about science. That shouldn't be surprising. They have terrible educations. After all, they studied premed in college and then went to a vocational school. MD/PhDs really ought to do better, but I suspect they do their doctoral work with faculty who come from the same under-educated group.
The problem is one of prestige. You cannot walk into a group of doctors and tell them why they're dumbasses and what to change. They know, and society confirms for them, that they are the best educated people on the planet and smarter than anyone else. "Heck, just look at our paychecks," they might respond. "That proves it right there, doesn't it? Figure it out from that data point only -- you don't need a control group."
Spinal Fusions and Hormone Replacement Therapy (Score:4, Interesting)
Don't do either without doing your own research.
I recently read a book called "How Doctors Think" by a Hematologist associated with Harvard. He went around interviewing the top doctors in the country about their cognitive fuck-ups and triumphs.
Serendipitously, along the way, some doctors were more than honest with him about medical dogma, being lazy in keeping up with research and how drug company money/gifts influence medical opinions.
Spinal fusions have no proven clinical benefit for reducing pain. At all. It has been known for a long time, but financial interests keep that procedure being done.
Hormone replacement therapy has been proven to be potentially dangerous ( cancer ,etc ) but with negligible benefit. Aside from middle aged women & estrogen, I'm sure you noticed the new commercials trying to convince aging men to supplement with testosterone.
It is natural for testosterone levels to gradually decrease as men age. Testosterone levels fluctuate often and hard. If a doctor tells you that you have below normal testosterone asked to be tested again. Get off of the things that lower testosterone ( ie alcohol ) and do the natural things that boost like hard exercise ( lift weights ) for a few weeks. Then get tested again.
Re:For example (Score:1, Interesting)
Ketosis is often mistaken for keto-acidosis and the two are very different. The former is indeed your friend. The latter is not. Ketones are the substrates remaining after you utilize fat for energy. Ketosis is a desirable effect- ketones are largely preferred by the brain and heart for fuel and are what our bodies evolved to burn. They are produced in the liver from fatty acids, are water-soluble, and can provide the majority of fuel for bodily tissues.
Ketoacidosis is when ketosis goes on unregulated by the body and is an excessive buildup of ketone bodies in the blood. In a normal person, even in starvation, ketoacidosis will not occur. Your kidneys will filter excess ketones or your liver will stop gluconeogensis before this occurs in a normal, non-pathogenic person.
Hope that clears up the ketone/ketoacidosis problem for anyone. If you still contest this issue in your mind or cannot resolve disparate information you may have heard, just remember, in the hundreds of thousands of years we lived prior to the agricultural revolution, carbohydrates were scarce. We evolutionarily best suited to fat metabolism. Sugar and carbs (while a treat) wreaks havoc on our metabolism and makes us sick.
Re:I see your problem right there: "Science" (Score:3, Interesting)
"I just don't get what you are implying here. Do you think that Computer "scientists" are better scientists on average than doctors? because the answer is no."
No, computer scientist are not scientists at all.
Neither are doctors. Science classes generally don't prepare you for doing science. Chemistry, for example, most often does "experiments" in the lab which are actually demonstrations of observed phenomena. Stats classes do a far better job of preparing you to actually do science, but nothing prepares you better than actually doing it. Constructing experiments with human subjects is notoriously difficult. Doctors do (as we see with their publications) a terrible job. And they train their own. That's a bad system perpetuating itself.
"Science" classes don't generally prepare you to do science for exactly the reason you point to with your computer science comment. We call a lot of things science in modern society, but we don't understand what real modern science is -- the concept of the pursuit of truth without ever absolutely being able to prove what is. We just come closer and closer to the truth while disproving hypotheses. Society doesn't like that definition very much. Drug companies don't like it very much. (It's unsatisfying at a human level, certainly.)
"Your points about prestige might be valid, but i would counter with a very hard leek at yourself. We know VERY little about the human body, most of what we know is from observation, not deduction."
After looking very hard at myself, I deduce lots of things. But that's Francis Bacon's idea of science you're describing (circa 1600), not a modern one. It's also the common conception of science, and that's a problem. (It's also probably your hypothetical surgeon's idea of science, even though he's been told different, since A. most of what he reads and sees and uses is science ala bacon, and B. a surgeon is generally the same as a dermatologist, though he may have been a better student, had steadier hands, worked harder and did very different post-school work -- he still has the same foundations.) Deduction is only the first step and sometimes not even really a required one -- it's the further steps of experimentation with well designed experiments which eliminate theories and provide real knowledge.
Finding cures, by the way, isn't science, most of the time, anyhow. Eliminating cures generally is science. Medicine, as you make clear in your post, is about trying stuff, seeing what works, and running with it. It's also about the application of tech and experience. Doctors do all that. They do a fairly good job. The medical profession educates its own to do that. But it does a bad job of science. It's not the same thing.
I suspect you think that I think that doctors are stupid. I don't. I think they're very well educated in their trade. That's called vocational school -- modern universities have been slowly stuffed full of vocational school ideas in the 20th century. And no, I don't think they're our best and brightest -- they start off as our best students, certainly, but we fail them by allowing and even encouraging them, at least financially, to get an education which only prepares them for being a doctor in our current conception of what a doctor is. That's a self-perpetuating cycle that gives us what we have. And part of that is bad science.
It's much harder to get funding to do medical research without an MD than it is to do it with one. So in order to get funding, people who have real science backgrounds and understand science spend 6 years getting an MD before they can get funding to do what they wanted to do 6 years before, and during that time, they are surrounded by bad methodology which they need to resist picking up. Then the papers that they write go to peer review journals -- and if they're in medical journals, they're reviewed by people who love their Bacon. Guess which articles they're more likely to like?
Bad cycle. Bad science. I suspect you could do a lot
Re:Reality check (Score:4, Interesting)
For example, I'm proposing a model of cognitive dysfunction in patients with Parkinson's disease (for my dissertation) that flies in the face of what most people who deal with Parkinson's disease believe is the case. My dissertation is not complete so I cannot say if my findings match my hypotheses but preliminary data are encouraging.
My point is that we have certain set beliefs about the way things are and some people (myself included) have a hard time challenging those beliefs. What I've seen is that many researchers don't actually follow the scientific method and so at the core, their research has methodological weaknesses.