Reanalysis of Clinical Trials Finds Misleading Results 74
sciencehabit writes: Clinical trials rarely get a second look — and when they do, their findings are not always what the authors originally reported. That's the conclusion of a new study (abstract), which compared how 37 studies that had been reanalyzed measured up to the original. In 13 cases, the reanalysis came to a different outcome — a finding that suggests many clinical trials may not be accurately reporting the effect of a new drug or intervention. Moreover, only five of the reanalyses were by an entirely different set of authors, which means they did not get a neutral relook.
In one of the trials, which examined the efficacy of the drug methotrexate in treating systemic sclerosis—an autoimmune disease that causes scarring of the skin and internal organs—the original researchers found the drug to be not much more effective than the placebo, as they reported in a 2001 paper. However, in a 2009 reanalysis of the same trial, another group of researchers including one of the original authors used Bayesian analysis, a statistical technique to overcome the shortcomings of small data sets that plague clinical trials of rare diseases such as sclerosis. The reanalysis found that the drug was, as it turned out, more effective than the placebo and had a good chance of benefiting sclerosis patients.
In one of the trials, which examined the efficacy of the drug methotrexate in treating systemic sclerosis—an autoimmune disease that causes scarring of the skin and internal organs—the original researchers found the drug to be not much more effective than the placebo, as they reported in a 2001 paper. However, in a 2009 reanalysis of the same trial, another group of researchers including one of the original authors used Bayesian analysis, a statistical technique to overcome the shortcomings of small data sets that plague clinical trials of rare diseases such as sclerosis. The reanalysis found that the drug was, as it turned out, more effective than the placebo and had a good chance of benefiting sclerosis patients.
Decline Effect (Score:5, Informative)
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Exactly, like with the theory of evolution and intelligent design. If the same amount of time was put towards the proof of intelligent design as with the theory of evolution, the atheists wouldn't have a leg to stand on.
Well the Intelligent design people had a couple of thousands of years working on their story. The scientists only ~160 years since Darwin wrote his 'On the Origin of the Species'. So I'd say we should give the scientists another 1900 years before we make any conclusions.
Re: (Score:2)
Re:Wooah! (Score:5, Interesting)
Almost had me there article! Until you said the most evil words known to man... "statistical technique". AKA "bullshit"
Bayesian statistics is far from bullshit.
I suggest you read up on it [amazon.com].
You can do some really cool stuff with it.
Testing if a coin flip is fair. [vosesoftware.com]
Correct images. [opticsinfobase.org]
Filter spam [arxiv.org]
Re: (Score:2, Insightful)
Re: (Score:2, Informative)
The problem is the Bayesian analysis is far from conclusive.
100% Wrong
What it tells you is the probability that your hypothesis is correct given your evidence and your prior knowledge.
Re:Wooah! (Score:5, Insightful)
Re: (Score:1)
Since people interpret confidence intervals as credible intervals (and they are usually close to same for a uniform prior), any problem with "uninformed" bayesian analysis is shared by those using frequentist techniques in practice.
Re: (Score:2, Funny)
Frequentists vs. Bayesians [xkcd.com]
Re: (Score:1)
The problem is the Bayesian analysis is far from conclusive. What it does point to is that the clinical trial needs a larger sample size. Sample sizes that are too small are useless.
Conversely if the sample size is too large, the cost of trials, typically a few to a few tens of millions of dollars, will go up significantly. The reflex answer to that is "so what, Big Pharma can pay for it!", except that many new therapies are developed by small companies, especially drugs for conditions with a small patient sample. For example, the condition used as an example, systemic sclerosis, has an incidence of about 1 in 100,000 in the US; it's estimated about 50,000 people in the US suffer fro
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
Try implementing that in practice on actual analysis of data.
I do it every day. Works perfectly.
An infinitely wide distribution function literary mean 1. F(x) = 1.
Do you need an example.
I have some data
x y
0.188334 2.08939
0.400133 2.26874
0.723409 2.31389
0.172104 2.00783
0.430118 2.28716
0.245059 2.03828
0.0494027 2.04421
0.221342 2.15249
0.911822 2.4686
0.461583 2.24511
I have a hypothesis that this data is linear. y = m*x + b. I want to know that probability that the parameters of m and b explain my data. I know nothing about m and b so I will set P(m)=1 and P(b)=1.
Re: Wooah! (Score:2)
Re: (Score:2)
But frequentist analyses aren't any more "objective," they just hide biases from view and include inductive biases that aren't even rationally compatible with any consistent state of belief.
With Bayesian analysis your starting point is out in the open and must be justifiable and defensible; analysts are accountable for their priors.
You can also, of course, examine what would follow from several different priors. This is much more straightforward than trying to shake the hidden biases in a frequentist model.
Re: (Score:2)
There's no universal method for constructing a prior. It's a big source of potential error.
Don't they just have to go into an monastery [wikipedia.org] or something like that?
Re: (Score:2)
Almost had me there article! Until you said the most evil words known to man... "statistical technique". AKA "bullshit"
--RAH
Not the usual way science is done (Score:5, Insightful)
Now that is an interesting observation! Mostly, in science, when someone does an experiment that supposedly proves a theory, the next step is to document and publish every detailed step. Only when a number of peers have replicated the results can they be accepted with any confidence.
Yet in clinical trials of new drugs, it seems, only a single trial is ever done. How did that ever get accepted as proper scientific evidence?
Re:Not the usual way science is done (Score:5, Insightful)
Whoops, I misunderstood the article for a moment there. If it's a matter of incorrect or misleading statistical analysis, that seems to be rife in studies of nutrition at least. Part of the problem may be that the same people develop a theory, conduct studies to test it, and do the statistical analysis on their numbers. Naturally, the numbers usually turn out to support their theory!
It might be safer if the three different activities were done by separate teams, with a "blind" system so no team knows who the other teams are. Thus the theory is developed by Team A, then studies/experiments to test it are created by Team B, and the number are analyzed by Team C. Thus Team C would have no idea what theory they were analyzing, or what might be the meaning of any correlations they found.
Re:Not the usual way science is done (Score:4, Insightful)
FYI there's already a blind system. Again - read people.
Re: (Score:2)
Drugs go through four phases of clinical trials [nih.gov], as required by the FDA.
That being said, the whole clinical trial process to get a drug approved by the FDA is pretty messed up. From how pharmaceutical companies are involved, how patients are (are not) qualified to participate, how adverse events [wikipedia.org] aren't necessarily properly documented, the list goes on.
Re: (Score:2)
Yet in clinical trials of new drugs, it seems, only a single trial is ever done.
That's not true at all. Generally, multiple trials are done and the most favourable results published. http://www.alltrials.net/ [alltrials.net]
Re: (Score:3)
There are lots of things that are messed-up about clinical trials, and the main reason for this is that they are VERY expensive to run. The problem is that you can't give somebody a drug unless their doctor is involved. Doctors make a lot of money, and clinical trials take a lot of time for them to participate in. So, if you want somebody who makes $500k/yr to spend 10 hours per week on a clinical trial, what is the only way to get them to cooperate? You have to pay them a LOT of money. Multiply this b
Re: (Score:1)
I think there's an underlying assumption to your comment that is incorrect, which is about the purpose of clinical trials. That's ok though, most people do not understand the purpose of the FDA and clinical trials.
The FDA's purpose is not to validate your science. That is not their mission, that mission is covered via journals/publishing/peer review processes. The FDA's job is in fact a marketing clearance organization.
The FDA was created because it was recognized that health is very complex that many pe
Re: (Score:2)
It didn't. Clinical trials are performed for exactly one reason: To get FDA approval. Not for scientific integrity, not for human safety. Purely for political purposes.
Its a crappy system. You hear all kinds of stories about companies just discarding the results of any trial that didn't prove their drug "worked" (because they're only required to provide a certain number of trial results. They're not required to provide the results of any trials they may have performed, or even note that they happened,
Selection bias? (Score:5, Insightful)
They looked at reanalyses that had already been done for other reasons, rather than doing their own reanalyses on randomly selected trials. It occurs to me that these trials may have been subjected to reanalysis precisely *because* there were doubts about the initial analysis.
Re: (Score:2)
So does that mean a re-analysis of the article on re-analysis leads to different conclusions than the original article?! HA!
But I have the sneaking suspicion that this re-analysis won't be published, which is a whole nother kind of selection bias can of worms.
AllTrials (Score:1)
Risks of Re-analysis (Score:4, Insightful)
Anytime you re-analyze data you run into this. [xkcd.com]
Think about it. There are a million ways you can analyze any dataset. There are millions of datasets out there to analyze. There are millions of people who can independently decide to go back and do a re-analysis.
So, the issue is that if somebody goes back and does a re-analysis and the results are boring, nobody publishes. However, if the results are controversial, it gets published. Since there are so many permutations, you're guaranteed to find something exciting.
This is why you're supposed to establish your methods BEFORE you collect the data, and then stick to the methods you established to analyze the data. Otherwise your 95% confidence turns into a more realistic 1% confidence.
In practice, though, I'm sure the initial analyses are just as prone to this kind of problem. It just gets REALLY bad when you look backwards.
statistics discussion (Score:4, Insightful)
There are many things wrong with clinical trials, but this isn't one of them. Both the original article and the reanalysis use valid statistical procedures and do not contradict each other. The original analysis didn't prove absence of an effect, it merely failed to show the existence of an effect. The new analysis shows that the drug is, in fact, more effective under some (weak, reasonable) a priori assumptions.
Whether to use statistical hypothesis testing (frequentist methods) or Bayesian analysis is a long-running debate in statistics and medicine. Both techniques are mathematically valid. Statistical hypothesis testing makes fewer a priori assumptions, which is why people have traditionally trusted it more and why it is widely taught and used in science. But over the years that people have come to realize that pessimistic assumptions can be harmful, such as when you continue clinical trials too long or reject the use of life saving drugs. Although I personally think Bayesian methods are a better way of analyzing the data, I think the debate over which methods to use is the way scientific debate and change should happen: slowly and with careful re-analysis and re-examination of data and experimental results.
Shedding light (Score:1)
This is why the meta-research regarding the safety and efficacy of GMO food organisms is so important. We are constantly told by the industry and their online astroturf army that there are "thousands" of studies showing the safety of GMOs, but it turns out they're basically the same shallow study 2000 times. The most disturbing findings have come from the kind of reanalysis that this story describes. But now, those studies get shouted down because supposedly it's "settled science".
Whenever there's money
Not so much with clinical trials (Score:1)
Drug companies and profit motive (Score:4, Interesting)
Let's compare two companies that depend on science - IBM and GlaxoSmithKline.
Let's say IBM discovers a new method of lithography for building microchips. They publish their results, and their results are replicated. More importantly, IBM gets a new, presumably better way of making microchips.
GlaxoSmithKline makes a new drug that treats a psychological illness. To some degree, because there are no objective physical tests for most psychological illnesses, the determination of effectiveness is made subjectively.
Both companies want the science to turn out right, because it makes them money. One of them has a much easier time massaging the results of any studies.
Re: (Score:2)
Well, in your analogy there is another big difference between IBM and Glaxo. In your example, if IBM does a faulty study that shows that a new way of making microchips is better, then the result will be that they spend a billion dollars on a new fab and it produces faulty microchips that nobody buys. In the other case, it is others who are relying on the results of the study to make a decision about whether to buy Glaxo's products. This is why it is much more important that drug trials be regulated. If
MDs do not grok statistics! (Score:2)
A real shocker. As basically no profession has a good grip on statistics except specialized mathematicians, it is no surprise so many are wrong or misleading.
Re: (Score:2)
... As basically no profession has a good grip on statistics except [statisticians], ...
There I fixed it for you.
Re: (Score:2)
You did fix exactly nothing. A statistician can also work in applied statistics and then the understanding is not required. So you basically BROKE the statement. Pathetic.
Statistics (Score:2)
Ultimate Argument for Reproducibility (Score:2)
There have been recent cries for reproducible results in science.
The scope is too limited.
There should be a cry for reproducible results in any research prior to its publication.
Long and short of it for researchers: if only you can get the results and conclusions, then the results and conclusions are not publishable.