Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Science News

Study Shows One Third of All Studies Are Nonsense 391

Posted by samzenpus from the 9-out-of-10-researchers-surveyed dept.
SydShamino writes "CNN has a report on new research to confirm claims made in initial, well-publicized studies. According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown. The study abstract is available."
This discussion has been archived. No new comments can be posted.

Study Shows One Third of All Studies Are Nonsense More

Study Shows One Third of All Studies Are Nonsense

Comments Filter:
  • My apology in advance for being a MC-person, but if 1/3 of the studies are inaccurate, which means this study can be 1/3 inaccurate, does it mean that the actual inaccuracy is 1/3 * 2/3 = 2/9 of all major studies are inaccurate?

    • This is just silly! Far too silly! (Score:4, Funny)

      by mister_llah ( 891540 ) on Wednesday July 13, 2005 @07:57PM (#13058782) Homepage Journal
      Well, it says 1/3 of the studies are inaccurate, so let us rank that on a percentage scale, say the study is inaccurate, we give them a 0 value.

      Accurate studies, lets say 100 (I know nothing is 100% accurate, and I know most studies even if they are somewhat accurate probably don't exceed 70% probability even in the specific environments they are enacted in, but lets just be over-generous since this whole thing is rather ridiculous anyway) ... *GASP FOR AIR* ... okay... so you have 66% chance of that particular study being at least somewhat accurate...

      Right? ... (rolls percentile dice)... OOOOHH! Man, rolled a 72, looks like I can't believe it.

      Rats.
    • Actually, this isn't a bad guess.

      The text of the article does not suppport the 1/3 bad claim exactly. Instead, it reports that 1/6 of initial reports are subsequently contradicted and another 1/6 are subsequently only weakly supported.

      Estimating from this range, the true number is probably somewhere in between, say 22.2% (=2/9) which is between 16.7% and 33.3%, or 24.5% which is the aveage of these?

  • Irony meter! (Score:5, Funny)

    by mister_llah ( 891540 ) on Wednesday July 13, 2005 @07:54PM (#13058743) Homepage Journal
    What... are you guys trying to blow up our heads?

    I think it is possible this is the most amusingly ridiculous piece of "legitemate" news I've read in awhile...

    Anyone got anything to beat it? Post it, I need to shock my brain a little more ;)

    • Re:Irony meter! (Score:5, Funny)

      by currivan ( 654314 ) on Wednesday July 13, 2005 @08:23PM (#13058972)
      Turkish shepherds look at dead sheep in the town of Gevas, near the city of Van, eastern Turkey, Thursday, July 7, 2005. First one sheep jumped to its death. Then stunned Turkish shepherds, who had left the herd to graze while they had breakfast, watched as nearly 1,500 others followed, each leaping off the same cliff, according to the Turkish media reported on Friday July 8, 2005. In the end, 450 dead animals lay on top of one another in a billowy white pile. Those who jumped later were saved as the pile got higher.
      Yahoo link [yahoo.com]
      • 450 dead is about one third of the 1500-strong flock that jumped. That is clearly numerologically significant, considering the article also is about 1/3 of something being effectively dead.

    • Re:Irony meter! (Score:4, Interesting)

      by Soul-Burn666 ( 574119 ) on Wednesday July 13, 2005 @08:24PM (#13058981) Journal
      Well there's that Guardian story [guardian.co.uk] about the transportation company suing 10 cleaning women for carpooling instead of using their overpriced and horrible service.

  • Nonsense! (Score:3, Insightful)

    by Speare ( 84249 ) on Wednesday July 13, 2005 @07:54PM (#13058750) Homepage Journal
    Since when is "inaccurate" or "overblown" nonsense? That's what science is: study something, make a theory, and just about dare someone else to prove it wrong, because that's what makes for a better theory.

    • Re:Nonsense! (Score:3, Insightful)

      by Nasarius ( 593729 )
      Exactly. Of course a good portion of studies, even when conducted properly, produce inaccurate results. That's the whole purpose of peer review; doing further research tends to filter out the bad stuff.
    • Many Americans seem to have a great deal of trust in science and scientific studies even though it is based on trust rather than personal experience and knowledge.

      I recall my church minister about ten years ago having a real problem with this and that he saw "trust" as a zero sum game. People were trusting science more than they were trusting the church.

      While I can't say what the motivation behind this study way, I wouldn't doubt for a moment if the same people who fight about evolution having no scientif

  • ERROR 0x381F (Score:4, Funny)

    by omarius ( 52253 ) <omar AT allwrong DOT com> on Wednesday July 13, 2005 @07:55PM (#13058756) Homepage Journal
    This study will cause an infinite loop..PLEASE SHUTDOWN IMMEDIATELY. 0x381F
    • This article seems also to have caused an infinite loop amongst our fellow slashdotters.

      Everyone is getting the irony and stating the same obvious joke

    • This study is actually alarming (Score:5, Interesting)

      by typical ( 886006 ) on Wednesday July 13, 2005 @11:16PM (#13060121) Journal
      Okay, I know that everyone likes throwing out wisecracks about the headline, which was ever-so-cleverly chosen by the article submitter, but consider the article for a moment.

      This is about the accuracy of clinical trial research. This is not about market research studies in the latest clothes fashions. Medicine is an extremely lucrative and risky field -- being associated with the group that pushes through the next Viagra can ensure that your family becomes the next Rockefellers. Your only opposition is the FDA (and the politicians that influence it, which are always hungry for money, which you have lots of).

      There is a tremendous amount of pressure on pharmaceutical researchers to produce favorable results. Let's say that you're a new, idealistic researcher who runs some tests on a new drug that your employer wants to market. Your tests show that our drug produces an increased rate of cancer? Well, been nice having you work here...bye. Bob down the hall has consistently gotten us much better results to feed to the FDA for approval. We really don't know how or why he gets better results, but he's definitely the man we want on the job. Sure, maybe twenty years down the road there will be some complaining, but *we didn't know*...*we did all our due dilligence and somehow our results just wound up showing that our drug was okay*.

      And even the more innocent "conclusive results" become suspect. A pharmaceutical doesn't want "inconclusive results", where "further tests are recommended". They have a bloody lifetime on the product ticking away, and a competition breathing down their neck. They want some scientist to sign off on this thing with a nice firm "Okay" or "Not Okay", or else what are they paying the guy for? He's not here to do ivory tower work -- he's here to serve the company, which is in the business of extracting savings from aging and achy baby boomers and subsidies paid for by their tax-paying children.

      What is being said is that a full third of examined clinical trials were essentially horseshit. This is really not a laughing matter.

  • Oblig. Futurama quote (Score:5, Funny)

    by dancingmad ( 128588 ) on Wednesday July 13, 2005 @07:55PM (#13058761)
    Zapp: Kiff we have a conundrum!
  • I think it states that most studies have inaccurate or overblown results not that 1/3 of studies are completely off.

  • Nonsense (Score:2, Insightful)

    by Coyote399 ( 666525 )
    Just because a study has inaccuracies doesn't mean the whole thing is nonsense.

  • Comment removed (Score:3, Funny)

    by account_deleted ( 4530225 ) on Wednesday July 13, 2005 @07:57PM (#13058787)
    Comment removed based on user account deletion

  • Falsifiability. (Score:2, Insightful)

    by Tackhead ( 54550 )
    > According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown.

    According to a recent study involving 100 clones based on DNA fragments of Karl Popper, a statistically significant number of the clones agree that this is pretty goddamn good result, considering that that's how science is supposed to work.

    You know - that silly process whereby you make a falsifiable claim, run an experiment, report your results, and encourage

    • Re:Falsifiability. (Score:5, Insightful)

      by cookie_cutter ( 533841 ) on Wednesday July 13, 2005 @08:38PM (#13059067)
      I agree with what you're saying, but that is not, exactly, the issue here.

      This isn't about hypotheses turning out to be false, it's about experiments which produce bad data, seemingly, at there release, supporting bad hypotheses.

      While even a good scientist can come up with wrong hypotheses, no good experimental scientist should be creating experiments which don't have proper controls to prevent them from drawing the wrong conclusions, nor should they be deriving conclusions based on an statistically insignificant sample.

      Arguably, the ability to design and implement properly controlled experiments and derive statistically significant results is what makes an experimental scientist and experimental scientist.

    • It's always very interesting to see reactionaries/creationists/evangelicals/luddites who don't understand the scientific method attempt to judge it using the framework of their own belief system, namely making the assumption that scientists must be like gods and their research therefore edicts that claim to come from on high, and thus, when those "edicts" don't hold true, it stands to reason that they are false gods, rather than The One True God that such people seek.

      I'm not sure that there's a way to ever

    • Re:Falsifiability. (Score:5, Insightful)

      by Spy Hunter ( 317220 ) * on Wednesday July 13, 2005 @09:41PM (#13059479) Journal
      Not exactly. Science is supposed to be a series of experiments designed to prove or disprove hypotheses. Having hypotheses disproved is of course a normal part of this process. However, having different experiments prove and disprove the same hypothesis is *not* a normal part of a healthy scientific process. It indicates either an incorrectly formed hypothesis or errors in experimental methods.

      Obviously errors are not completely avoidable because people are fallible; that's why we try to reproduce results and practice peer review. But I should think we ought to do better than having 33% of our supposedly "proven" hypotheses eventually disproved by subsequent experiments.

      Note that I'm not talking here about trivial things like Netwon's laws of motion being "disproved" by relativity. Relativity is more like a generalization of Newton's laws than a refutation, and that *is* a part of the normal scientific process. I'm talking here about medical studies which come up with conflicting results or the innumerable global warming studies that the scientific community can't make up its mind on (for example).

      • Re:Falsifiability. (Score:3, Informative)

        by Angry Toad ( 314562 )

        It indicates either an incorrectly formed hypothesis or errors in experimental methods.

        Or limitations of methodology. I think a lot of cutting-edge science tends wander along the edge of this problem - there may be an effect, but the available data is only barely sufficient to see it, and obtaining a statistically sound sample size would be uneconomical. Lots of good research ends up exploiting clever tricks to get around this kind of limitation - and sometimes falls prey to unforseen effects and influ

      • Re:Falsifiability. (Score:3, Insightful)

        by radtea ( 464814 )
        However, having different experiments prove and disprove the same hypothesis is *not* a normal part of a healthy scientific process. It indicates either an incorrectly formed hypothesis or errors in experimental methods.

        Nonsense. Science is carried out by human beings, and human beings make mistakes. Healthy processes accomodate that fact.

        Publishing the results of those mistakes, honestly and fully for the critique of others, is part of the scientific process. Having those mistakes corrected by later
      • More than half of these fallacies can be attributed to the medical field.

        Most importantly, the subscription-drug companies.

        I dare you to look it up and prove different. Thus is the basis of science, as mentioned earlier.

        Money motivates science just as much as any other. Look at asbestos, used to be it was approved by the FDA. Decades following, was proved harmful by too many studies to ignore.

        Just a suggestion, give ANY "scientific fact" at least a decade before you believe it to hold any water ;)

      • Politics and money drives the FUD. (Score:4, Interesting)

        by TapeCutter ( 624760 ) on Thursday July 14, 2005 @02:00AM (#13060815) Journal
        "...innumerable global warming studies that the scientific community can't make up its mind on (for example)." - Bad example, climate scientists "know" the planet is warming and also why it is warming, but fossil fuel politics creates an enormous amount of FUD in an attempt to make you and me think the scientists are contradicting each other and basically haven't got a clue. A similar thing occured when medical scientists said tabacoo was bad for your health. Incredibly some of the same "researchers" who "proved" tabacco was harmless have also been involved in "proving" climate scientists are wrong.

  • Better science education required. (Score:5, Insightful)

    by Colin Smith ( 2679 ) on Wednesday July 13, 2005 @08:00PM (#13058809)
    The fact that they get away with it is a shame. It's even worse when they have an influence on government policy. Ugh.

    Lots of people can't think of a good reason to do science, maths and statistics at school. Well, a bloody good reason is so you can prevent the wool being pulled over your eyes.


  • No doubt the study of studies is itself of dubious quality.

    Personally I would have expected that the Pareto principle (20% of anything is the important part) or Sturgeon's law (90% of everything is crap) would have been the operational forces here.

  • What a nice paradox this story presents. "I am lying."

  • The topic is stated toooo broadly! (Score:3, Interesting)

    by mister_llah ( 891540 ) on Wednesday July 13, 2005 @08:01PM (#13058829) Homepage Journal
    The way the topic makes this sound, this is some sort of blatantly obvious study...

    it isn't, really :)

    It is about the effectiveness of interventions... if you skipped over it, its worth a perusal to a skim... at the very least... but it would seem to me that the whole thing has lead to almost no positive conclusion itself... with 44% of the experiments being replications and 24% unchallenged... the 66% really don't seem to have much value... ... so it's kind of... ambiguous...

    Ahhh, academic research... only there can you get paid well to tell us absolutely nothing...

  • Studies are wrong because... (Score:5, Funny)

    by Alpha27 ( 211269 ) on Wednesday July 13, 2005 @08:02PM (#13058838)
    four out of three people have problems with fractions.
  • Questionable studies like Second-Hand Smoke Is Bad [blogspot.com] or Drink 8 Glasses of Water a Day [detnews.com] (does beer count?)

    • Re:More nonsense studies: (Score:5, Funny)

      by Anonymous Coward on Wednesday July 13, 2005 @08:15PM (#13058931)
      Drink 8 Glasses of Water a Day (does beer count?)

      Depends. One would have to calculate the water content of beer versus the rate of dehydration produced by the alcohol content. Following through, one would conclude that 8 glasses of beer would fall short of the goal of 8 glasses of pure water, with the only recourse being to drink more beer.

      This, kids, is a practical demonstration of how to make science work for you.
    • > Drink 8 Glasses of Water a Day (does beer count?)

      If you drink beer with 6% alcohol, you will only have drunk 7.52 glasses of water. So you would have to drink another half glass to compensate for the alcohol.

      Cheers!
  • Since there is a 33% chance that this study itself is wrong, I hereby scoff and dismiss it!

  • Overblown (Score:3, Funny)

    by nick_davison ( 217681 ) on Wednesday July 13, 2005 @08:04PM (#13058849)
    According to the new study, about a third of all major studies from the last 15 years were subsequently shown to be inaccurate or overblown.

    The actual figure turns out to have been 26.4% - much closer to 1/4 than 1/3.

    • Re:Overblown (Score:4, Informative)

      by uhoreg ( 583723 ) on Wednesday July 13, 2005 @08:35PM (#13059052) Homepage
      Where did you get 26.4% from? They looked at 49 studies. 45 of them reported that intervention was effective. Of those 45, 7 were subsequently contradicted, and 7 were found to report stronger effects. So that's 14 that are "inaccurate or overblown". 14/45 (which follows the percentages given in the abstract) is 31% (which is pretty close to 1/3). If you want to do 14/49 (the total number of studies they looked at), that gives you 28.6%.

  • Oh yeah! (Score:2, Funny)

    by JeiFuRi ( 888436 )
    I did my own study and it shows that one half of all studies are nonsense.
  • It's true. Those letters your aunt, grandmother, and that complete stranger who thinks you are their college friend because you have the same initials are the most reliable source of scientific, health, and political information:

    Aspartame causes MS. Andy Rooney has definitive proof that the founding fathers wanted prayer in the classroom. Microwaving food in plastic causes cancer. George Carlin lays out why librals are dumb. Bill Gates wants to buy you a car for forwarding this email.

    All true!

    Stefan
  • Most published studies include statistics claiming something like a 95% (or higher) chance of being correct. I've always thought that was bogus and it looks like someone finally got around to proving it.
  • /me rolls 4d6

    7

    didn't make the saving throw, I don't belive the story.
  • This CNN story is antiscience nonsense. The report might have something useful to say about the 45 pharma studies that were analyzed - if its results can be repeated by another analyst, and its methodology is sound (assumedly vetted by the JAMA publishers). But it doesn't say anything about "science". Science doesn't give final answers. Of course many studies are subsequently shown to be inaccurate: the reproducability of the experiment tests the theory, and more data eventually converge on the actual stat

  • No shit (Score:3, Insightful)

    by grasshoppa ( 657393 ) on Wednesday July 13, 2005 @08:07PM (#13058883) Homepage
    When companies can buy reports and studies to say whatever the fuck they want them to say (*cough*microsoft*cough*), of course they are going to be bullshit.

    Who's surprised by this? Seriously.

  • Science by press conference (Score:5, Insightful)

    by jokestress ( 837997 ) on Wednesday July 13, 2005 @08:07PM (#13058884)
    Sadly, this has become a cottage industry for less scrupulous publicity-hungry hacks in academia and elsewhere. Think Clonaid or cold fusion. Come up with some hasty conclusion and make a grand announcement before the data is available or has been tested by others.

    Even worse are the lazy journalists who report it. After a New York Times piece last week claimed bisexual males were "lying" [nytimes.com] based on results from a highly questionable study, I reminded their editors of this excellent piece Blinded by Science [cjr.org] in Columbia Journalism Review.

    This kind of sloppy reporting is perfect for lazy journalists-- it's a three-for-one deal. They get to break the news, and then later they have a second story when real experts point out the flaws, and a third when the people finally get discredited. More evidence of the shameful state of journalism in this country.

  • ..and one third of the replies to this article will be lame jokes about this study being nonsense.
  • how many studies are produced by creationists, or the oil lobbies' climatologists.
  • At work we did a study using state DOE data, we showed a correlation between high scoring (we review Juvenile detention programs by visiting them and scoring them on different benchmarks) programs and students rate of returning to school and graduating. Seems pretty obvious, and the results were as we expected. Odd thing is another team did a followup study with their own methods, both of which seemed to make sense, except they were studying kids released a year later than the ones we studied. They produced
  • Physicist 1 to physists 2: One in three scientists believe their peers fabrictate research.
    Physicist 3, to herself: He's making that up...
  • This study, found in the scholarly journal Duh...
  • This guy [enterprisemission.com] alone probably tipped the scales considerably...
  • Which 1/3 does this study fall under?

  • I wonder if _this_ study is in the 1/3 of them that are found to be nonsense.

    My head hurts.

  • Well if you look at all the studies for and against global warming you quickly realize that there are hundreds of studies out there that have to be overblown garbage. How else can you explain so much contradicting data?
  • See subject

  • Missing the point (Score:5, Informative)

    by kebes ( 861706 ) on Wednesday July 13, 2005 @08:30PM (#13059016) Journal
    I feel like alot of posters are not understanding what the study is... this is probably because the abstract (or, if you have access, the actual article) is much more meaningful than the CNN report.

    First, notwithstanding the many good jokes about a self-referrential study that will proven to be exaggerated, this study specifically checked whether highly cited clinical studies had claims that were later contradicted or softened due to other research. This study was not claiming that 1/3 of all scientific studies published were wrong in some way. It's worth noting that doing clinical research is very difficult, and that the error bars will always be quite large. It's also important to keep in mind that sometimes clinical research may be unduly influenced by financial pressures... and that clinical research undergoes very heavy scrutiny.

    So having 1/3 of all clinical studies be later contradicted should not make us worry that clinical research is being done wrong. We should be happy that so much verification occurs, that any erroneous conclusions will (probably) be checked again. One line from the CNN article rings true:
    Experts say the report is a reminder to doctors and patients that they should not put too much stock in a single study and understand that treatments often become obsolete with medical advances.

    I think that should be the take-home message for the casual reader. Science is doing its job of verification, but people need to stop jumping to conclusions (or worse, changing their life habits) based on the results of a single study. The results need to be double-checked. The study may have been a fluke, or have flaws, or the data may have been manipulated. Whatever the reason, we should not trust single experiments, especially where human lives are at stake!

    Having (partially) read the JAMA article, I think their result is sobering and useful. It really shows how intense the competition is in that field (which leads both to people making exagerrated claims, but also alot of pressure to dis-prove other's claims and get at the "right answer").

  • winner-take-all vs. long-tail effects (Score:5, Insightful)

    by G4from128k ( 686170 ) on Wednesday July 13, 2005 @08:31PM (#13059026)
    Connectivity - global media, the internet -- have created a winner-take-all world that drives both the creators of studies and the reporters of studies toward hyperbole. If someone wants their 15 minutes of fame, they need to do (or appear to do) something spectacular. When attention is a scarce resource (because of an explosion of applications/demands for attention), then it drives people toward excessive behavior in crafting and reporting the results of studies.

    At the same time, I wonder if the long tail efect [wikipedia.org] means that an increasing number of once-obscure, high-quality studies are being discovered, read, and used by an increasing number of people. Those that do create unbiased studies may not get much popular press, but they do become more widely read due to Google.

    Ultimately, we seem to be floating in a rising tide of both good and bad studies. Perhaps the ratio of studies is being biased toward the bad (winner take all) but the ratio of impressions -- the numbers of times that good studies have been accessed -- has actually improved due to long-tail effects.
  • I don't think the problem is a surfeit of nonsensical studies, so much as a misrepresentation of them by the media. All the time, studies are reported as news. Yet even a well-carried out, accurate study does not prove anything. Studies by their nature must be corroborated to be meaningful at all. When a new study comes out, it doesn't mean that we know anything new. It means that somebody suspects something which may in the long run turn out to be true. The problem is that the latter is not sensational eno

  • AP Statistics (Score:2, Interesting)

    by michaelzhao ( 801080 )
    I can personally relate to this. While in AP Statistics, we practiced our mad stat skillz on the real world. We were encouraged to bring in newspaper clippings of studies and experiments and see if they were statistically sound. Most weren't. The most common form of bias in those studies were known as negative response. In this form of bias, only people with negative, strong feelings reply to the study question.

    I am willing to bet that the CNN study is correct in it's assumption that most studies are incor

  • Obligatory Monty Python quote (Score:3, Funny)

    by Flashpot ( 773365 ) on Wednesday July 13, 2005 @08:57PM (#13059166)
    Ordinary decent people are sick and tired of being told that ordinary decent people are fed up with being sick and tired. Well I'm certainly not, and I'm sick and tired of being told I am!

  • Post is misleading (Score:3, Insightful)

    by Ingolfke ( 515826 ) on Wednesday July 13, 2005 @09:24PM (#13059353) Journal
    The research was done for MEDICAL studies, not tech studies, or animal habitat studies, or psychological studies, or sociological studies... only medical studies. Nowhere in the title or the post's main body is this mentioned. This is very poor reporting of the news. It is misleading. The study also only measured studies from 1990 to 2003. That's 13 years not 15 years!

    Word to the wise, don't trust the press at face value. Expect sources, preferably cited and available for you to review, and check your facts before you buy into whatever the press happens to be reporting today.

  • Parent Title Is Complete Crap (Score:3, Informative)

    by DynaSoar ( 714234 ) * on Wednesday July 13, 2005 @10:44PM (#13059910) Journal
    All that studies report is their results. They don't gaurantee other studies will find the same thing. If they did, there'd be no reason to have replications. This is a basic part of doing science.

    And who's to say the replications aren't the ones that missed the mark?

    1/3 right
    1/3 wrong
    1/3 we have no idea what the answer means.
    That's what I was told to expect from research in my first semester of grad school. Not from reading it -- from DOING it.

    They really should teach science in school. Not the just areas of science, but the subject of science itself.

Slashdot Top Deals

I have hardly ever known a mathematician who was capable of reasoning. -- Plato

Close