Study Shows One Third of All Studies Are Nonsense

SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."

Missing the point

[kebes]

I feel like alot of posters are not understanding what the study is... this is probably because the abstract (or, if you have access, the actual article) is much more meaningful than the CNN report.

First, notwithstanding the many good jokes about a self-referrential study that will proven to be exaggerated, this study specifically checked whether highly cited clinical studies had claims that were later contradicted or softened due to other research. This study was not claiming that 1/3 of all scientific studies published were wrong in some way. It's worth noting that doing clinical research is very difficult, and that the error bars will always be quite large. It's also important to keep in mind that sometimes clinical research may be unduly influenced by financial pressures... and that clinical research undergoes very heavy scrutiny.

So having 1/3 of all clinical studies be later contradicted should not make us worry that clinical research is being done wrong. We should be happy that so much verification occurs, that any erroneous conclusions will (probably) be checked again. One line from the CNN article rings true:
Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.

I think that should be the take-home message for the casual reader. Science is doing its job of verification, but people need to stop jumping to conclusions (or worse, changing their life habits) based on the results of a single study. The results need to be double-checked. The study may have been a fluke, or have flaws, or the data may have been manipulated. Whatever the reason, we should not trust single experiments, especially where human lives are at stake!

Having (partially) read the JAMA article, I think their result is sobering and useful. It really shows how intense the competition is in that field (which leads both to people making exagerrated claims, but also alot of pressure to dis-prove other's claims and get at the "right answer").

Re:Overblown

[uhoreg]

Where did you get 26.4% from? They looked at 49 studies. 45 of them reported that intervention was effective. Of those 45, 7 were subsequently contradicted, and 7 were found to report stronger effects. So that's 14 that are "inaccurate or overblown". 14/45 (which follows the percentages given in the abstract) is 31% (which is pretty close to 1/3). If you want to do 14/49 (the total number of studies they looked at), that gives you 28.6%.

Re:Science by press conference

[krbvroc1]

The slashdot abstract is incorrectly summarized. This study is reported in the Journal of American Medicical Association and relates to 'clinical studies'. The summary infers a wider group of studies.

Re:The topic is stated toooo broadly!

[flosofl]

[...]its worth a perusal to a skim

GAH! Someone needs to look at the definition for "perusal"

Perusal [reference.com] - To read or examine, typically with great care.

And don't get me started on your use of "its" (its=possesive it's=contraction of "it is")

I know, I know... Mod me +5 Pedantic. I've had a long day, and that felt good. :)

Re:Falsifiability.

[Angry Toad]

It indicates either an incorrectly formed hypothesis or errors in experimental methods.

Or limitations of methodology. I think a lot of cutting-edge science tends wander along the edge of this problem - there may be an effect, but the available data is only barely sufficient to see it, and obtaining a statistically sound sample size would be uneconomical. Lots of good research ends up exploiting clever tricks to get around this kind of limitation - and sometimes falls prey to unforseen effects and influences (which I suppose more or less falls under your "errors" column, but I think it's important to mention this slightly aberrant case and put an asterisk beside it).

Parent Title Is Complete Crap

[DynaSoar]

All that studies report is their results. They don't gaurantee other studies will find the same thing. If they did, there'd be no reason to have replications. This is a basic part of doing science.

And who's to say the replications aren't the ones that missed the mark?

1/3 right
1/3 wrong
1/3 we have no idea what the answer means.
That's what I was told to expect from research in my first semester of grad school. Not from reading it -- from DOING it.

They really should teach science in school. Not the just areas of science, but the subject of science itself.

Re:This study is actually alarming

[Anonymous Coward]

Wow, you really have no idea how clinical studies for FDA approval work, now do you? And yet, unsurprisingly, there are enough "geeks" here that believe your bullshit to mod it up to +5