Study Shows One Third of All Studies Are Nonsense 391
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
Mathematically Challenged (Score:2, Insightful)
Nonsense! (Score:3, Insightful)
Studies inaccurate but not completely bogus (Score:2, Insightful)
Nonsense (Score:2, Insightful)
Falsifiability. (Score:2, Insightful)
According to a recent study involving 100 clones based on DNA fragments of Karl Popper, a statistically significant number of the clones agree that this is pretty goddamn good result, considering that that's how science is supposed to work.
You know - that silly process whereby you make a falsifiable claim, run an experiment, report your results, and encourage others to add to the store of scientific knowledge by attempting to falsify your original hypothesis?
Better science education required. (Score:5, Insightful)
Lots of people can't think of a good reason to do science, maths and statistics at school. Well, a bloody good reason is so you can prevent the wool being pulled over your eyes.
Re:Mathematically Challenged (Score:2, Insightful)
The text of the article does not suppport the 1/3 bad claim exactly. Instead, it reports that 1/6 of initial reports are subsequently contradicted and another 1/6 are subsequently only weakly supported.
Estimating from this range, the true number is probably somewhere in between, say 22.2% (=2/9) which is between 16.7% and 33.3%, or 24.5% which is the aveage of these?
Re:Nonsense! (Score:5, Insightful)
Re:more like (Score:5, Insightful)
The fact is, yes, statistics can be misused. So can every other field of study. But used right, statistics are a tremendously powerful way to understand our world, and often reveal information that can't be obtained any other way. And believe me, nobody gets more peeved at statistics abuse than statisticians do.
But that's okay, pal. Just keep on making fun of things you don't understand. The smart people of the world will keep on working, keep doing things that make your and everyone else's life better, whether you know it or not.
No shit (Score:3, Insightful)
Who's surprised by this? Seriously.
Science by press conference (Score:5, Insightful)
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" [nytimes.com] based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science [cjr.org] in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
Re:Nonsense! (Score:3, Insightful)
Re:The Statistical Methods in Most Studies are Fla (Score:3, Insightful)
Re:Obviously flawed (Score:2, Insightful)
It is!
No, please don't call my bluff!
What are those cuffs [telegraph.co.uk] about?
Screw you all! SCREW YOU ALL!
Re:Obviously flawed (Score:1, Insightful)
Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" [nytimes.com] based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science [cjr.org] in Columbia Journalism Review.
This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.
winner-take-all vs. long-tail effects (Score:5, Insightful)
At the same time, I wonder if the long tail efect [wikipedia.org] means that an increasing number of once-obscure, high-quality studies are being discovered, read, and used by an increasing number of people. Those that do create unbiased studies may not get much popular press, but they do become more widely read due to Google.
Ultimately, we seem to be floating in a rising tide of both good and bad studies. Perhaps the ratio of studies is being biased toward the bad (winner take all) but the ratio of impressions -- the numbers of times that good studies have been accessed -- has actually improved due to long-tail effects.
Re:Falsifiability. (Score:5, Insightful)
This isn't about hypotheses turning out to be false, it's about experiments which produce bad data, seemingly, at there release, supporting bad hypotheses.
While even a good scientist can come up with wrong hypotheses, no good experimental scientist should be creating experiments which don't have proper controls to prevent them from drawing the wrong conclusions, nor should they be deriving conclusions based on an statistically insignificant sample.
Arguably, the ability to design and implement properly controlled experiments and derive statistically significant results is what makes an experimental scientist and experimental scientist.
Re:Obviously flawed (Score:1, Insightful)
"Controversy and uncertainty ensue when the results of clinical research on the effectiveness of interventions are subsequently contradicted."
Effectiveness of interventions! This only looked at studies in which interventions were involved. NOT ALL STUDIES.
To hyperbolize in the same manner..... I think the author is the dumbest person in the entire universe.
Re:Falsifiability. (Score:3, Insightful)
I'm not sure that there's a way to ever really reframe the worldview of people socialized in such a way to help them understand the secular, methodical, aggregate-dialectic nature of of the scientific method.
You'd think that the results it produces (i.e. the very computers, electricity, television, and telephone used by so many reactionaries to try to preach the ills of the scientific method) would go some distance toward demonstrating to them the empirical utility of the method for instrumental-rational gains (regardless of the merits of such), but no--they remain oblivious to the obvious paradox.
Re:Falsifiability. (Score:3, Insightful)
To claim that science is false or that scientists are the same as priests is to completely ignore a history of socially powerful, yet materially impotent priests and a contrasting history of socially impotent, yet materially powerful scientists.
I'm playing with you a little bit now, but you get the point.
Post is misleading (Score:3, Insightful)
Word to the wise, don't trust the press at face value. Expect sources, preferably cited and available for you to review, and check your facts before you buy into whatever the press happens to be reporting today.
Re:Obviously flawed (Score:3, Insightful)
This appears to be particularly frequent in more abstract (non-maths) sciences like environment. (I once had lectures on the topic where the speaker cited stats that did not match the notes and were inconsistent across presentations.)
Re:Falsifiability. (Score:5, Insightful)
Obviously errors are not completely avoidable because people are fallible; that's why we try to reproduce results and practice peer review. But I should think we ought to do better than having 33% of our supposedly "proven" hypotheses eventually disproved by subsequent experiments.
Note that I'm not talking here about trivial things like Netwon's laws of motion being "disproved" by relativity. Relativity is more like a generalization of Newton's laws than a refutation, and that *is* a part of the normal scientific process. I'm talking here about medical studies which come up with conflicting results or the innumerable global warming studies that the scientific community can't make up its mind on (for example).
Re:/. posting shows 99% of /. postings are bullshi (Score:3, Insightful)
the main reason (Score:2, Insightful)
And then it could segue into something roughly analagous to the debates over for-sale closed source software and collaborative information-sharing free software. Could a news reading public be persuaded to actually become critical reporters and "share" news freely? Could it replace the expensive and established profit motive design of "news" as we know it today?
Some might say blogging is at least an attempt in that direction.
Re:Nonsense! (Score:5, Insightful)
The Slashdot summary of this particular article is more than a little misleading and sensationalist.
Wait a minute... (Score:2, Insightful)
Re:Falsifiability. (Score:3, Insightful)
Nonsense. Science is carried out by human beings, and human beings make mistakes. Healthy processes accomodate that fact.
Publishing the results of those mistakes, honestly and fully for the critique of others, is part of the scientific process. Having those mistakes corrected by later researchers who have the benefit of seeing what earlier researchers have done and the luxury of contemplating the problem from the perspective of "How do I improve on this?" rather than the vastly more difficult, "How do I do this the first time?" is also a healthy part of the scientific process.
The day no one ever publishes anything for fear it might not be the perfect, irrefutable experiment is the day that science is dead.
Re:more like (Score:5, Insightful)
However, statistics are not determinative. This is a mistake I've heard from both laymen and experts. The fact that, according to what's known (and factored in to the calculation) an event is 99.999999999% likely to happen... well, that doesn't mean it will happen. Really, it doesn't matter how unlikely your statistics demonstrate something to be, it won't prevent the unlikely from happening.
In fact, it's demonstrable from statistical analysis that we should expect tremendously improbable events to happen quite often, and that the chances of the most probable outcomes to occur at every instance is an incredibly unlikely outcome as time stretches on.
So statistics are an interpretive tool, not an answer. Statistics alone cannot tell you what will happen, they can't tell you what has happened, and they certainly cannot tell you what should happen. And all these comments you're talking about, I think they come from a valid frustration borne from sloppy reporting telling us "scientists have discovered that 75% of" this and "they now know that 25% of" that outside of any meaningful context.
And what's the likelihood that all these percentages are correct? What's the margin of error, and what's the margin of error's margin of error? Certainly the people telling us these "facts" (reporters) have no idea.
Re:Nonsense! (Score:5, Insightful)
At one time the NEJM was considered a "gold standard" in publishing medical research. IMO, that's no longer true. I do believe that journals are no longer as careful about what they publish. [Rhetorical question- why did Marcia Angell leave as editor of the NEJM?]
While we do expect "science" to eventually verify or refute claims made in scientific studies, I do not remember such a high percentage of studies being "overturned", as the lawyers would say. [Of course I am susceptible to "recall bias.
Along with journals more willing to print research, I do believe that the quality of research itself has worsened. [I've seen first hand some of the "fun and games" that take place in academia. That's one reason I'm no longer there!]
In addition, the public's hunger for news about health and lifestyle and the media's need for sensationalism have fueled a news feeding frenzy. I doubt that the public or the news media are really capable of judging the worth of clinical studies. I don't think the public or the media understand the medicine or science involved nor do they actually know how to evaluate research on scientific, methodological or statistical grounds.
I get really peeved when I see some study touted on TV that reports a 10% reduction in some disease supposedly caused by modifying some risk factor. Without seeing the confidence interval bouding this estimate of risk, I'm not willing to say the effect is real. As a rule of thumb, I was taught to view with skepticism any study that does not halve or double the relative risk attributed to a risk factor.
Re:Falsifiability. (Score:3, Insightful)
Most importantly, the subscription-drug companies.
I dare you to look it up and prove different. Thus is the basis of science, as mentioned earlier.
Money motivates science just as much as any other. Look at asbestos, used to be it was approved by the FDA. Decades following, was proved harmful by too many studies to ignore.
Just a suggestion, give ANY "scientific fact" at least a decade before you believe it to hold any water
Re:nice (Score:2, Insightful)
It could lead to an escalation in hysteria, racial hatred and so forth from otherwise-rational people who believe the end of the World is nigh, or something.
Case in point: Following the London attacks there was a bomb scare in Birmingham (about 120 miles North) which resulted in 20,000+ people being evacuated from the city centre. It turned out to be a firework. Taken in isolation I'm sure the response to something like this would be a lot more measured, but coming so soon after the events in London everyone (including the Police) reacted in a typical knee-jerk fashion.
I guess my point is that there is obviously never "a good time" for a plane to crash into a neighbourhood, but there is certainly a less dramatic time for it to happen.
Please define religious nut (Score:3, Insightful)
While there are a large number of people who reject facts and reason due to their a priori commitment to a religious beliefs, there are a great number who do the same whose religion is science itself.
That is to say, preconceived notions and personal bias prevent many so-called scientists from acknowledging facts and realizing that their pet theories are baseless.
As an example, I offer Carl Sagan. Here was a man who made a nice living talking about extraterrestrial life. Is there ANY evidence of extraterrestrial life? Is there ANY science that supports it? After all, the best that the SETI institute has is Drake's equation which at best merely multiplies speculation upon speculation.
Is SETI science? Perhaps, but Sagan's beliefs and public discussions were based on fantasy and hope rather than fact.
My point is this: Bias appears in religion and in the name of science. Science has dirty hands, too.
Remember, power tends to corrupt, regardless of world view. I'd be willing to bet that a similar book could be written demonstrating horrible abuses of human rights where science was allowed to 'progress' unchecked by morality.
Finally, it is important to note that much of science has been advanced by people with strong religious convictions. Pascal, Pasteur, Lister, Knuth, Kelvin, Joule, Carver, Bacon, Boyle, and many many others. Strong religious conviction is NOT the antithesis of scientific advancement, as demonstrated by the legacy of those I listed above and I could list many more.
Respectfully,
Anomaly