Forgot your password?
Science News

Most Science Studies Tainted by Sloppy Analysis 252

Posted by ScuttleMonkey
from the devil-is-in-the-details dept.
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
This discussion has been archived. No new comments can be posted.

Most Science Studies Tainted by Sloppy Analysis

Comments Filter:
  • by Pojut (1027544) on Tuesday September 18, 2007 @12:24PM (#20654241) Homepage
    They can study my taint anytime they want. /Karma

  • Yup. (Score:3, Insightful)

    by Estanislao Martínez (203477) on Tuesday September 18, 2007 @12:24PM (#20654251) Homepage
    And they are routinely reported sensationalistically in the media, and most of you people who are reading this right now swallow it all hook and sinker.
  • by Penguinisto (415985) on Tuesday September 18, 2007 @12:25PM (#20654253) Journal
    Insert politically charged science topics, point to 'em as examples, and launch into a stupid flamefest over it all in 3... 2... 1...


  • as a phd student (Score:3, Insightful)

    by KeepQuiet (992584) on Tuesday September 18, 2007 @12:27PM (#20654295)
    all i can say is "duh!". Everybody, being under pressure of "you have to publish", publish whatever they can. Sad but true
    • As a postdoc, I'd be interested to know what field you're in and where you study. If you actually believe what you just said, get out now. We don't want people who're so susceptible to fraud in science.
      • by everphilski (877346) on Tuesday September 18, 2007 @12:56PM (#20654985) Journal
        It is fairly common knowledge that 3 things factor into tenure (in this order): (1) being published (2) bringing funding into the university and (3) teaching.

        1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.

        2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.

        3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).

        The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.

        SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know ...
        • Number 3 is something akin to citizenship: participating in meetings, bringing in guest speakers, etc.
          • by Wiseazz (267052)
            I think you could cover a whole range of activities for #3 by simply saying "Ass kissing".

        • The whole "more publications" thing is baloney. 15 mediocre articles will NOT get you tenure at a competitive university. They are not that stupid. I've seen this many times. The flip side is that many articles with solid data published in decent but not high profile journals is also unfortunately not always enough to get tenure. So ya, there are many issues, but every department is different, and every field is also. I can't imagine you're in biology, because 15 articles in 6 years is bullshit. Maybe CS or
  • Sensationalist... (Score:5, Insightful)

    by posterlogo (943853) on Tuesday September 18, 2007 @12:28PM (#20654317)
    It is way off the base to say that "most published research findings are wrong". It is often the case that data analysis and interpretation for particular aspects of a research project (like 1-2 figures in a 7 figure paper) are up for vigorous debate. The scientific community can, in the long run, converge on very robust ideas, and drop those that are flimsy. To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.
    • by posterlogo (943853) on Tuesday September 18, 2007 @12:36PM (#20654509)
      Furthermore, this epidemiologist primarily studied medically-related publications, and in fact focused mostly on high-profile research that make broad claims, or relied heavily on statistics to support a conclusion. Many research publications at the cell/molecular level do not rely on subtle statistical comparisons to prove a point. This guy is singling out research that is based heavily on correlations (like people with x, y, z are more likely to get a, b, or c diseases). He is only an expert in his own field, and I don't think he is qualified to judge every level of scientific publication, but he certainly doesn't mind the media attention.
      • by Miraba (846588)
        Oh, if only I had a modpoint...
        Medical research != all scientific research; it's much more prone to errors due to how it's performed and analyzed. I hope your point doesn't get buried amongst all the charges of corruption.
        • Sloppy analysis (Score:3, Insightful)

          by benhocking (724439)
          So, is what you're basically saying is that this study was tainted by sloppy analysis?
          • by Miraba (846588)
            No, the article in the WSJ is tainted by sloppy analysis. The paper itself (a metareview) shows that the authors recognized that their own biases could be a problem. From the abstract: "Two evaluators independently extracted data with a third evaluator arbitrating their discrepancies."
    • To misleadingly imply that most research is wrong, which is exactly what the post suggests, is just poor interpretation of flimsy data, ironically.

      Irony police, your analysis?
    • Re: (Score:3, Interesting)

      by budgenator (254554)
      In the Army they taught us that when doing an inspection and the paper work was written in two different colors of ink, and the last bit was in pencil, and the paper itself looked like it had been caught in the rain and folded up and carried in someones back pocket for three days its probably authentic so be suspicious if the paper work is too neat and clean. I see science the same way, if there is no arguments about the data or conclusions, everybody is talking about the majority and consensus, i again get
      • Consensus (Score:3, Interesting)

        by huckamania (533052)
        Before there is consensus on an issue, there is contention. Before contention there is no theory at all. Only at the the consensus phase can a majority of 'scientists' be correct. During the contention phase, there is the old theory that didn't include the new theory or which may in fact preclude the new theory. In both cases, if the new theory is correct, then the old theory was not. At least part of the pile of peer reviewed papers for the old theory can now be viewed as incorrect.

        What's interesti
  • by BWJones (18351) * on Tuesday September 18, 2007 @12:28PM (#20654321) Homepage Journal
    It should be noted that "medical research" (epidemiology, clinical studies etc...) is very different from basic research (mechanisms, pathways, etc...) and the threshold for acceptance in journals that cover basic research is much higher than that for medical journals. i.e. There is significantly higher oversight and peer review criticism over basic research than there is medical research and the two fields should not be confused.

    • by brteag00 (987351) on Tuesday September 18, 2007 @12:40PM (#20654605)
      It's not just medical research. The scientific community works like any other community: the greater the implications, the greater the scrutiny, attempts to replicate, etc. The Huang embryonic stem cell study is a great case-in-point: the image-manipulation fraud was uncovered because of the vast number of researchers looking at the micrographs he published. (That sounds familiar, doesn't it: "Many eyes make all bugs shallow.") Global warming has many, many people working on models, taking ice cores, doing other analysis. Of course, the vast majority of published research isn't reported in Science [] or Nature [], and so it doesn't get as much exposure. That's why around here (the University of Wisconsin), it's standard practice that if your work depends on someone else's result, you first replicate her experiment and make sure you get the same result. (If you can't, you write a letter to the appropriate publication making note of your inability to replicate the result.) This means that eventually the mistake gets uncovered, and your research doesn't get burned because someone else has been sloppy.
      • Re: (Score:3, Insightful)

        by Otter (3800)
        That's why around here (the University of Wisconsin), it's standard practice that if your work depends on someone else's result, you first replicate her experiment and make sure you get the same result. (If you can't, you write a letter to the appropriate publication making note of your inability to replicate the result.)

        Out of curiosity:

        1) What is the usual failure rate for replication?

        2) Do the letters routinely get published?

        3) You just do that for work you're following up with experiments, not for ever

        • Re: (Score:2, Interesting)

          by brteag00 (987351)

          1) What is the usual failure rate for replication?

          2) Do the letters routinely get published?

          3) You just do that for work you're following up with experiments, not for everything you cite, right?

          Unfortunately, I'm not in a hypothesis-driven lab [], so I can't speak to any of these from direct experience. I know that I routinely see such letters published (frequently as "technical comments"), and I know that I go to seminars and routinely see people get raked over the coals for not having verif

    • by flynt (248848)
      Also, within medical research, a clear divide must be appreciated between randomized, controlled, clinical trials and epidemiology. A well-run clinical trial is about as good of an experiment as you can do. Patients and doctors remain blinded to the actual treatment so biases are not introduced.

      Epidemiology is almost always done retrospectively, and while it may have its uses, there are *always* going to be possible confounding variables when patients are not randomized before receiving a treatment.

      So ple
  • by sarahbau (692647) on Tuesday September 18, 2007 @12:30PM (#20654351)
    How do we know the study that shows that most studies are tainted isn't tainted?
    • by temcat (873475)
      Damn, you beat me to it :-)

      Yeah, general negative statements tend to negate themselves.

  • by Archangel Michael (180766) on Tuesday September 18, 2007 @12:30PM (#20654375) Journal
    According to my research, most studies involve about 84% error rate due to flawed statistical analysis caused by people pulling statistics out of their arse. The other 16% are flawed due to NOT actually pulling statistics out of their arse.
  • My experience (Score:3, Insightful)

    by Zelos (1050172) on Tuesday September 18, 2007 @12:31PM (#20654389)
    This was certainly true in my experience. When I did a review of mathematical methods in my area a while back, most papers had basic calculation errors, missing information that made reproducing the work difficult or impossible, and they all used carefully selected examples to show their work in the best light.
  • by Gallowglass (22346) on Tuesday September 18, 2007 @12:32PM (#20654417)

    And one of the first rules is, "Never take a single study as proof of anything! Wait till the results are replicated before you even think of moving to a conclusion."

    The major problem is really poor reporting on science research. The news media routinely blazon some **NEW * Scientific * Discovery!!!**. Then you read the story and somewhere around the 10th paragraph you might see that this is based on only one study - and oftentimes even before peer review.

    Every scientists knows this. It's a shame the public doesn't. They wouldn't worry so much.

    • I agree. The problem with that approach is two-fold.
            1) Funding agencies will fund new research, but will not fund research to confirm previous research.
            2) The competition for funding, tenure, etc has led to an ever increasing specialization of expertise. Gone are the days of the gentleman scientists where anybody with a tower could replicate Galileo's experiments.

    • The sample sizes are often small in medical/health studies and human beings have a lot of extra variables that are hard to control. The best thing to do is to wait for the meta-study, where someone analyzes all the studies that relate to an effect. After many studies have been done, do they agree? Do they appear to have been well done? Adding the studies together creates a larger sample size and hopefully averages out some of the variation due to flaws in the method.
    • by drfireman (101623)

      The major problem is really poor reporting on science research.
      This is indeed a major problem. But it doesn't have much to do with this article, which discusses poor analysis by scientists of their own data, more or less. When a scientists can collect data, fail to understand what can be learned from it, misreport it, and have that misreporting compounded by more misreporting by journalists, then you have... well, the world as we know it.
  • by packetmon (977047) on Tuesday September 18, 2007 @12:34PM (#20654467) Homepage
    The secret is out! Investigators are looking into whether or not millions of scientists have been using modified versions of SCIgen [] for their work. The FBI and Department of Termpaper Security have acknowledged the investigation but declined to speculate on the alleged ties between SCIgen and grammar terrorists citing a new law just passed by pResident Bush which allows warrantless underwear tapping.

    Authorities are also investigating the connections between Malda, Bush Laden, Bill Gates, Dvorak and Borat [] SCIgen is a program that generates random Computer Science research papers, including graphs, figures, and citations. It uses a hand-written context-free grammar to form all elements of the papers. Our aim here is to maximize amusement, rather than coherence.
  • o the irony (Score:4, Funny)

    by Anonymous Coward on Tuesday September 18, 2007 @12:34PM (#20654475)
    /. commentors commenting on sloppy submission about sloppy analysis

    pot, meet kettle
  • "Most science..." (Score:5, Insightful)

    by Otter (3800) on Tuesday September 18, 2007 @12:34PM (#20654477) Journal
    In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.

    His work seems to focus on population genetics and epidemiology, which is notorious for having unreproducible claims due to a combination of uncorrected multiple testing, publication bias and statistical incompetence. This "gender and genes" is a perfect example: someone does a study, finds nothing, slices and dices the data until he gets p = 0.04 for females or Asians or smokers and publishes his breakthrough finding. I'd have been surprised if he hadn't found almost all of those to be wrong.

    If you look at more in-vitro molecular biology and biochemistry work, I doubt if nearly as high a percentage of it is clearly "wrong", although quite a bit of it is worthless.

    • Re: (Score:3, Interesting)

      by AJWM (19027)
      Doesn't surprise me. Most people who go into the "fuzzy" vs the "hard" sciences do so because they're not good at or don't like math. (Yeah, I know, broad generalization.)

      And as math goes, statistics can be pretty darn counterintuitive.

      (I speak from experience - I worked for a few years in a university computer center's "academic support group", where among other things I help faculty and grad students with running statistical analysis packages. Some of the experimental designs were pretty bad, too.)
      • Re: (Score:3, Interesting)

        by Otter (3800)
        Most people who go into the "fuzzy" vs the "hard" sciences do so because they're not good at or don't like math.

        Actually you need more math for a PhD in psychology or sociology than you do for molecular biology, unless you're in a really math-heavy specialty.

        The biologists, being generally smart, can usually pick up what they need (math, programming, equipment building) on the fly. The problem is that, as you say, statistics frequently is counterintuitive.

    • by eli pabst (948845)
      To some degree that's why it's now harder to publish genetic epidemiology papers, especially things like association studies. Many journals are now requiring that you show some kind of functional effect of a mutation that's associated with a disease; gone are the days when you can publish something just because p0.05. And while many people assumed that the inability to replicate results was due to flawed statistics/methodology, we're also finding that there are large differences between different human po
  • by Edward Ka-Spel (779129) on Tuesday September 18, 2007 @12:37PM (#20654547)
    I just read the article and checked his statistics. He did his numbers wrong.
  • by cduck (9999) on Tuesday September 18, 2007 @12:46PM (#20654753)
    I am not a scientist.

    That being said, it's my understanding that most scientists work off of grants, and those grants fund novel research. Replicating results is of obvious importance in validating those results, but doing so seems at odds with the funding mechanisms that are the reality for what I would believe to be most researchers.

    Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?

    (As rhetorical as that might have sounded, I actually welcome those with first-hand experience to respond to it)
    • I'll talk about my master's thesis work. I was designing a guided bullet to be shot out of a 40mm cannon with a linearized guidance system and a pack of squibs. The combination of a linearized guidance system and a controller hooked up to the squibs would cancel out any pointing errors and initial guidance errors of the gun+RADAR system as the bullet was in flight, and then hopefully hit the target. I wrote a 6DOF simulation to model all this and that was the basis for my thesis.

      Now, to answer your questi
    • Re: (Score:2, Insightful)

      by Anonymous Coward
      I am a scientist (polymeric materials).

      Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?

      You are correct that no grant money is specifically allocated for reproducing other's results, nor for generating "null results" (showing that somethings isn't the case, e.g. that a particular methodology *won't* work). This is a problem, because it means that some important findings that are "uninteresting" don't get studied (or, worse, the data exists but never

  • Bad Dates (Score:3, Informative)

    by HTH NE1 (675604) on Tuesday September 18, 2007 @12:47PM (#20654769)
    Quoting []

    EA Wallace Budge is one of the great authorities on Egyptology, but his work is badly out of date, and was actually never all that good. If nothing else, he has a tendency in his translations to treat Egyptian theology as monotheistic in the model of Christianity. In the Stargate movie, Daniel Jackson says 'I don't know why they keep printing him': The simple answer is that the copyright is expired, so it's cheap, and his name still shifts copy.
  • It must be so... (Score:3, Insightful)

    by Actually, I do RTFA (1058596) on Tuesday September 18, 2007 @12:53PM (#20654919)

    "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."


    Since the criterion is that the claim is published, someone had to find the study new and interesting. Most new ideas are going to be wrong, especially true the more significant it is. After all, how many crackpot theories were postulated between Newtonian and Relativistic physics? On the other hand, most things easily verifiable, etc, are too obvious to me considered new and interesting. Note, while I find this interesting, I did not come up with this idea. Some economists published a similar study over a year ago postulating this as a reason. Of course, it's probably wrong.

  • The secret shame of the scientific community is that statistical analysis is the foundation of all good research but few Ph.D programs offer more than a single semester worth of train in the subject. Truth is, training in statistical analysis should start in grade school but I doubt that will happen any time soon. One solution is dropping high school and college requirements of calculus and replace them with a year of statistics, which would be useful to more students.....
    • Problem with that is most of the people who need statistics will need some knowledge of calculus. Maybe they won't be explicitly composing integrals or differential equations but they will need a good understanding of rates, areas under the curve, gradients, etc... but I do agree statistics in high school beyond mean/median/mode would be useful.
    • I agree. I was a physics major and never took a single statistics class. They would throw in some probability and statistics in other classes, but it always seemed to be something you were just supposed to pick up on your own. You would just be taught random tools as the situation arose, but never really learned what tool to apply to what situation.

      I think life science majors got a much better education in statistics than physical science majors.

      I think every science major should required to take a class i
  • by call -151 (230520) * on Tuesday September 18, 2007 @12:56PM (#20654981) Homepage
    There are a lot of different attitudes about the role of the anonymous referee, in different fields and in different settings. In computer science and mathematics, where most of my publications are, the role of the referee depends upon a number of things. A few comments relevant to my disciplines:

    • The responsibility for correctness lies with the author, not the referee. It is good if the referee spots problems but it is not the obligation of the referee to certify that every last detail is correct.
    • Often, the primary responsibility of the referee is to comment on the importance, priority, relevance and how much interest there is in the work.
    • In the CS world of conference refereeing (as opposed to CS journals) there is often absurd time pressure. Articles/abstracts are due at midnight local time on some date, so things are typically hastily written, and referees must review things in a very short timeframe and practically never get a chance to check things carefully. As far as I am concerned, the conference publication model in CS is terribly broken. There have been some calls for reform, but those have been coming for at least the last 10 years or so and over that period it's gotten worse, not better.
    • In math, it can take a year for a referee to work through something techical, so the process is slow.
    • Typically, referees are uncompensated for their work. Some people take their refereeing duties seriously, and some do not. Generally, those who do a good job in a timely fashion are asked more often to referee more things, which is not exactly a reward.

  • [The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.] ...
    [No one knows how much shoddy research is out there.]

    I'm not sure what the point of this article is besides fear mongering. The goal of most scientific research is to prove a set of assertions - and sure this set may not be fully encompassing or comprehensive - but you've got a model and you try see what fits - and its not always exact.

    Take for example the recent debacle /wrt Steve McIn
  • Hmmm... what fraction of news published and edited in reputable journals contains factual errors? That would be another interesting story.

    But come on... 90% of everything is crap. This is no more or less true in medical research, which is a fraction of the $50 billion total spent on research. OMG that's like "One MILLLLIIION DOLLLLARS". That's like 0.5% of GDP so don't be surpised when it's bunk, it's drop in the bucket compared to development costs. If you haven't figure this out you're still a little naiv
  • Cargo Cult Science (Score:5, Insightful)

    by ObsessiveMathsFreak (773371) <{ten.mocrie} {ta} {kaerfshtamevissesbo}> on Tuesday September 18, 2007 @01:10PM (#20655265) Homepage Journal
    People need to realise that a lot of those calling themselves scientists are not really scientists at all. They don't apply the scientific method. They massage data regularly. They misapply statistics constantly. They don't subject their theories to falisfiability []. They waffle, hand wave, engage in rhetoric, and generally do just about everything except an honest to goodness, old fashioned solid, scientific experiment.

    Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists []. They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.

    It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.

    Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.

    There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.
    • Re: (Score:3, Interesting)

      by Rimbo (139781)

      The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. ...

      There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is.

      I agree with you. There are two reasons why your method does not happen more often.

      The first is that failure in science is perceived to be a failure of reason. Almost

  • by rumblin'rabbit (711865) on Tuesday September 18, 2007 @01:12PM (#20655327) Journal
    Universities are pumping out PhD's at a prodigious rate. As a manager of R&D, I've interviewed and hired more than my share. Virtually all say they want to do research.

    Here's my problem. Only a fraction (I'm guessing 1 out of 5) are actually capable of doing good research. The rest are competent employees for developing other people's research into useful products, but aren't terribly original thinkers, nor show a lot of initiative, nor show the rigour and clarity of thought one wants to see in a researcher.

    Frankly, when I "unleash" employees on open-ended problems without much guidance, the majority soon begin to flounder.

    There is nothing wrong with getting advanced degrees, but many then feel they are obliged to do original research when in fact they really aren't up to it. This may be one reason why the quality of papers isn't where it should be.
  • Is this Wall Street Journal article on the credibility of (medical) scientific research an invitation to scientists to publish in Nature that modern economic theory is grounded in the flawed assumptions that individuals will make choices based on the most desirable outcome in every instance? Or their insistence on ignoring that an economic system is inherently finite, and thus unlikely to settle into an optimal state? Or how about the time scale of settling, which in many cases would be longer than human
  • yeah but... (Score:2, Funny)

    by Kelexel (1158529)
    [Speaking to self...]
    A scientific study published that most scientific studies are wrong... therefore there is a good change of it being wrong.... Which means that most scientific studies are right... But if most studies are right then this one is also right... which means...
    c.. an.. t take .... this...
    [Head explodes]
  • "Torture statistics enough, and it will admit to anything."
  • by Durandal64 (658649) on Tuesday September 18, 2007 @01:32PM (#20655735)
    This guy's main beef appears to be with medical studies and other sciences which rely heavily on statistics (sociology, psychology and the other wannabe-sciences). This is not surprising, to be honest. Statistical analysis isn't difficult, but I've known many social science students. They consider statistics to be extremely advanced and have no other mathematical background. As a result, they don't have a very deep understanding of how to mathematically model a system. Naturally, this will lead to bogus conclusions and incompetent analysis work. Medicine has a similar problem, albeit on a smaller scale. Most of the time, statistical analysis will yield correlations, but they won't tell you anything about the mechanism behind what you're seeing, which is what's important in science.

    I'd expect the rate of error for physics experiments to be much lower than that of, say, sociology.
    • Re: (Score:3, Insightful)

      by Bluesman (104513)
      "Statistical analysis isn't difficult"

      I'm not sure what depth of statistical analysis you're talking about, but I've found statistical analysis to be exceptionally difficult when applied to Electrical Engineering, and I know I'm not alone here -- it was by far the most difficult post-graduate class at my school.

      I can confidently say that nobody who's graduating with me has a complete grasp of all of the statistical tools we were taught. Enough to get by, yes, but most of the things are extremely counterint
  • These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis.

    Said the WSJ editorial board...

  • I'm sorry (Score:3, Funny)

    by Sloppy (14984) on Tuesday September 18, 2007 @01:54PM (#20656185) Homepage Journal
    I won't analyze again.
  • by Goldsmith (561202) on Tuesday September 18, 2007 @01:57PM (#20656235)
    The Wall Street Journal headline is a tautology. (Note that he's not talking about scientific misconduct, only honest mistakes, incorrect analysis or experimental design which could be improved.)

    There are almost no areas of science we're "done" with. The most recent paper on a subject almost always points out where previous papers have gone wrong. Thus, the previous papers have some mistake such as a miscalculation, poor design or incomplete analysis. If you pick any paper published in a peer reviewed journal this month, there's a very high probability that at some point in the future it will be amended or improved by some other paper.

    What Ioannidis *has* shown in his recent reports is that in genetics, not enough people are publishing on the same subjects. There are not enough "other papers" out there to check on the previous ones. The result is that papers which in other fields would be recognized as needing improvement are instead treated as the final word.
  • Summary (Score:3, Funny)

    by slapout (93640) on Tuesday September 18, 2007 @02:03PM (#20656371)
    So, to sum it up for the Slashdot audience:

    • Global warming isn't happening
    • Pluto IS a planet
    • Han shot first
  • by fasta (301231) on Tuesday September 18, 2007 @03:07PM (#20657637)
    Putting aside for a moment the question of whether Genetic Association Studies - the focus of the research paper - are representative of "Most Science", the article does not say the analysis is invariably sloppy, it says it is often mistaken. For genetic association studies, this is not surprising, since it is very difficult to publish a negative result. So, small studies that show a statistically significant relationship are published, but small studies showing no relationship are not. Then, when larger studies are done, the small studies that had the "significant" relationship because of a fortunate or unfortunate set of samples is not confirmed. Indeed, this is what the research article points out; if your threshold for statistical significance is 0.05, then you will report that a chance relationship is significant once in 20 experiments. But, if you can't publish the 19 negative experiments, then lots of chance results get published.

    But Dr. Ioannidis has a very narrow definition of science - he only includes statistical studies that use p 0.05 as a threshold for significance. There are, of course, lots of papers that do not show p-values - the purification of a protein, the determination of a genome sequence, the identification of a new fundamental particle. In many cases, p-values are not provided because they are not considered informative - something that happens when the p-value is much much much less than 0.05 (I like my p-values less than Avagadro's constant. With that p-valuep, I think most of my results are correct.)

    And, of course, the WSJ misses all of this. The point of the research paper is that you can do everything right, and still be mislead with marginal p-values (0.05). Not sloppy, just not significant enough. We could, of course, require more stringent values, but then we would miss the genuinely rare, but important results.

    As the research article points out, results that are reproducible are, in fact, quite likely to be correct. It is perhaps useful to distinguish between science as a paper and science as a process. Most results that stand up to scientific scrutiny over a period of years (that any one cares enough about to validate), are (probably) correct. In some disciplines, which rely heavily on modest thresholds for statistical significance, many results cannot be confirmed.
  • by Mutatis Mutandis (921530) on Tuesday September 18, 2007 @03:12PM (#20657725)

    I have worked in a biotech / pharmaceutical environment for over five years now, and I don't trust the average medical researcher or biologist to accurately calculate the weight of a kilogram of stuff. I would say that their data analysis is habitually poor, if I were not convinced that it is actually habitually awful.

    I have been trying to change this for five years. My success in this has been such that it contributed strongly to my recent decision to start searching for another job. The reality is that biomedical researchers simply do not believe in doing mathematical analysis of data properly. They consider it an eccentric habit, forgivable but socially objectionable, like smoking. By common consensus, it is considered much too complicated to expect that any of them can be expected to understand. Your average biologist is innumerate to the nth degree, and proud of it.

    I blame their education, which seems to stress naive and antediluvian (excuse the word) analysis practices, if at all. I have seen course materials which in their expression of basic mathematical formulas, betrayed that they had been left unchanged since the days when people used slide rules and logarithmic tables for calculations. Most of their other training is strictly qualitatively, not quantitatively, and focussed more on memorizing that on understanding.

    If necessary, they will find a crutch to help themselves to stumble along: Find a paper that defines a formula that looks relevant, and then fill in the numbers. They would not bother doing their own analysis, or trying to understand how the calculation works or whether it is relevant at all. The notion that a good statistical analysis of mathematical modeling can actually contribute to the scientific understanding of an issue, is well beyond most of them.

    I am frankly, sick and tired of their attitude, and I still have to work with these people every day. And in my experience, my colleagues are actually better than most. I strongly suspect the WSJ is correct on this one.

  • by tfoss (203340) on Tuesday September 18, 2007 @04:05PM (#20658777)
    Some Epidemiological Claims of Sex Differences for Genetic Effects Not Replicated.

    This is a *very* small number of claims from a subsection of a single field of one small bit of science. Tarring all of science based on some potentially dubious epidemiology is badly out of line. It would be like claiming that since some spinach has made people sick, all food is unsafe to eat. Absurd.

    Epidemiology itself has a bit of a reputation of having a hard time finding really solid effects, partly because the effects that are measured are frequently multi-variate with lots on confounding effects, partly because you need huge numbers to have very much analysis power, partly because such studies are generally more observational then experimental. This guy has published a bunch of papers in the past arguing (and presenting models [] for) exactly this kind of problem. He comes up with the logical (if rather obvious) suggestions that amongst others: 1. Smaller studies are less likely to be true. 2. Smaller observed effects are less likely to be true. 3. The greater the financial interests there are in the study, the less likely it is to be true. 4. The "hotter" a topic is, the less likely a study is to be true. Largely these are no shit, sherlock kinds of things.

    So, to sum up, there are lots of epidemiological claims in published articles out there that might not be right. This represents neither a new idea, nor a meaningful comment on anything but epidemiology.

  • by geekoid (135745) <> on Tuesday September 18, 2007 @04:52PM (#20659671) Homepage Journal
    getting a paper published is the very first step in peer review, not the final word.
    So yes, this might be a problem, but when other peers review it the problems are likely to get pointed out.
    A peer review paper isn't a paper that HAS been peer reviewed, it one that is being peer reviewed.

    Yes, I know that war redundant, but people for get to all to often.

    Another reminder - Scientist live to disprove hypothesis and theories.
  • by Jerry (6400) on Tuesday September 18, 2007 @06:07PM (#20660825)
    That was the title of a NOVA film in 1998.

    `Abstract: This video examines the troubling question of scientific fraud: How prevalent is it? Who
    commits it? And what happens when the perpetrators are caught? Factors contributing to "bad science"
    include sloppy research, personal bias, lack of objectivity, "cooking and trimming", "publish or perish"
    pressure, and outright fraud. The limits of peer review and other quality control systems are discussed.'

    The results of the study determined that 48% of all published data was fraudulent. The data was trimmed, cooked or outright falsified. Some cases made famous by public exposure were analyzed.

    While recieving a lot of lip service from the establishment science, the two government researchers who made the report were reassigned to worthless tasks in isolated areas. One was sent to shuffle papers in Alaska, IIRC. So much for whistle blowers, even government whistle blowers.

    In the last 19 years it seems nothing has changed. Besides this latest report how can I tell? Simple. The news is filled with stories of drugs being recalled because they are more dangerous that the problems they are supposed to treat. How would they ever have gotten on the market in the first place if their FDA "studies" weren't rigged? And you don't wonder about the revolving door policy between Pharmaceutical employees and FDA employees? Corporate influence in research is as corrupting as Microsoft influence in ISO standards voting.

    What really burns me is that MUCH of our basic research is done at academic institutions by professors funded by government grants, i.e., tax payers. But, thanks to the best congress that money can buy (because most of them have been bought off) OUR research is "monetized" (sold to special interests) for pennies on the dollar. These interests then reap HUGE license profits for decades. To make matters worse, many of the "special interests" are the very academic researchers who were paid to do their work. Having discovered key facts, without reporting them, they resign academia and begin a corporation to capitalize on what we paid them to learn.

    IF we had a congress worth what they are paid there would be a law which prohibits recipients of gov grants, or their families relatives, or former business associates to personally benefit from what they learned using that grant money for a period of 15 years. Secondly, the ONLY corporations which should be allowed to receive IP licenses from the gov should be NON-PROFITS, whose board, management or employees cannot include the professor or his family or relatives.

    Another thing that this recent study shows is what the NOVA film revealed: Peer-review is worthless for vetting research. Replication is worthless for vetting research. Obviously, personal integrity is also a worthless indicator of research quality.

"Card readers? We don't need no stinking card readers." -- Peter da Silva (at the National Academy of Sciencies, 1965, in a particularly vivid fantasy)