Forgot your password?
typodupeerror
Science

Why Published Research Findings Are Often False 453

Posted by samzenpus
from the race-to-publish dept.
Hugh Pickens writes "Jonah Lehrer has an interesting article in the New Yorker reporting that all sorts of well-established, multiply confirmed findings in science have started to look increasingly uncertain as they cannot be replicated. This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology and in the field of medicine, the phenomenon seems extremely widespread, affecting not only anti-psychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants. 'One of my mentors told me that my real mistake was trying to replicate my work,' says researcher Jonathon Schooler. 'He told me doing that was just setting myself up for disappointment.' For many scientists, the effect is especially troubling because of what it exposes about the scientific process. 'If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved?' writes Lehrer. 'Which results should we believe?' Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to 'put nature to the question' but it now appears that nature often gives us different answers. According to John Ioannidis, author of Why Most Published Research Findings Are False, the main problem is that too many researchers engage in what he calls 'significance chasing,' or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. 'The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,'"
This discussion has been archived. No new comments can be posted.

Why Published Research Findings Are Often False

Comments Filter:
  • Hmmmmm (Score:5, Interesting)

    by Deekin_Scalesinger (755062) on Sunday January 02, 2011 @12:30PM (#34737664)
    Is it possible that there has always been error, but it is just more noticeable now given that reporting is more accurate?
    • Re:Hmmmmm (Score:5, Insightful)

      by Joce640k (829181) on Sunday January 02, 2011 @12:43PM (#34737754) Homepage

      Maybe it's just the the truths being presented in the article are the sort of 'truths' that are hard to measure 100% objectively. Whenever results have a human element there's always the possibility of experimental bias.

      Triply so when the phrase "one of the fastest-growing and most profitable pharmaceutical classes" appears in the business plan.

      Fortunately for science, the *real* truth usually rears it's head in the end.

      • Re:Hmmmmm (Score:4, Insightful)

        by gilleain (1310105) on Sunday January 02, 2011 @12:50PM (#34737824)

        Maybe it's just the the truths being presented in the article are the sort of 'truths' that are hard to measure 100% objectively.

        Alternatively, this article is almost unbearably stupid. It starts off heavily implying that reality itself is somehow changeable - including a surfing professor who says something like "It's like the Universe doesn't like me...maaan".

        This is just a journalistic tactic, though. Start with a ridiculous premise to get people reading, then break out what's really happening : poor use of statistics in science. What was really the point of implying that truth can change?

        • Re:Hmmmmm (Score:5, Informative)

          by wanax (46819) on Sunday January 02, 2011 @01:59PM (#34738342)

          Well, passing over for the moment the likes of determinism and ecological psychology, I think you're mistaken in direct studies of human behavior. There are a number of very subtle effects, that when run through the non-linear recurrent processes of the brain can lead to significant behavioral changes (ie. the demand effect [wikipedia.org]). While some of these were touched on lightly in the New Yorker article (about blinding protocols and so on) there are second order effects that are impossible to control. A psychologist who does a large subject pool experiment needs funding, which to get generally requires pilot results. These results are exciting to the psychologist and their lab, they're more motivated, have higher energy, probably are interacting better with the subjects (or the technicians running the subjects, if they have enough money to blind it that deeply), more motivated people are going to produce different behaviors than less motivated people. If the blinded study is positive and appears significant, it may become a big thing.. but by the nth repetition of the original experiment by another lab to verify they understand the protocol, the lab might be completely bored with the initial testing and the result disappears, essentially a variation of the Hawthorne effect [wikipedia.org] (which has itself been disappearing). That may well mean that the effect exists in certain environments but not others, which is an ultimately frustrating thing to classify in systems as complex as human society.

          It essentially boils down to the fact that we're all fallible, social beings that interact with the environment, rather than merely observing it. Whether you want to say that this adds correlated but unpredictable noise to any data analysis that is not being appropriately controlled for (but can be), or is fundamental limit on our ability to understand certain aspects of the world, at our current level of understanding it does rather seem that there is a class of experiments in which a scientist's mental state affects the (objective) results.

          • by hitmark (640295)

            So basically a kind of psychological uncertainty principle?

          • does rather seem that there is a class of experiments in which a scientist's mental state affects the (objective) results.

            Or rather, it does seem that some social scientists have much poorer standards than their colleagues in the "hard" sciences. That seems a much simpler explanation than inventing a new "class" of experiments.

            There's nothing new about errors that creep in due to the scientist's mental or physical state. Look up the theory of experimental errors [wikipedia.org], which was invented two h

        • Re:Hmmmmm (Score:5, Interesting)

          by bughunter (10093) <bughunter&earthlink,net> on Sunday January 02, 2011 @02:50PM (#34738698) Journal

          Start with a ridiculous premise to get people reading, then break out what's really happening

          Welcome to corporate journalism. And corporate science.

          If there's one useful thing that 30 years of recreational gaming has taught me, it's this: Players will find loopholes in any set of rules, and exploit them relentlessly for an advantage. Corrolaries include the tendency for games to degenerate into contests between different rulebreaking strategies and the observation that if you raise the stakes to include rewards of real value (like money) then the games with loopholes attract players who are not interested in the contest, but only in winning.

          This lesson applies to all aspects of life from gaming, to sports, business, and even dating.

          And so it's no surprise that when the publishers set up a set of rules to validate scientific results, that those engaged in the business of science will game those rules to publish their results. They're being paid to publish; if they don't publish, they've "lost" or "failed" because they will receive no further funding. So the stakes are real. And while the business of science still attracts a lot of true scientists -those interested in the process of inquiry- it now also attracts a lot of players who are only interested in the stakes. Not to mention the corporate and political interests who have predetermined results that they wish to promulgate.

          What was really the point of implying that truth can change?

          To game the system, of course. The aforementioned corporate and political interests will use this line of argument now, in order to discredit established scientific premises.

          • by gilleain (1310105)

            Heh. This explanation appeals to me. It reminds me a little bit of an article called "Playing to Win" that talks about 'scrubs'. If you haven't read it, a scrub will complain when an experienced player seems to exploit loopholes but is actually just playing the game.

            I think that the two situations you mention (science and journalism) can feel a bit like games sometimes. Perhaps the players are becoming confused as to the real purpose of the game. In the case of science, it is to advance knowledge; not to ad

      • Re:Hmmmmm (Score:5, Insightful)

        by causality (777677) on Sunday January 02, 2011 @01:28PM (#34738078)

        Maybe it's just the the truths being presented in the article are the sort of 'truths' that are hard to measure 100% objectively. Whenever results have a human element there's always the possibility of experimental bias.

        Triply so when the phrase "one of the fastest-growing and most profitable pharmaceutical classes" appears in the business plan.

        The pharmaceutical industry is easily one of the most corrupt industries known to man. Perhaps some defense contractors are worse, but if so, then just barely. It's got just the right combination of billions of dollars at play, strong dependency on the part of many of its customers, a basis on intellectual property, financial leverage over most of the rest of the medical industry, and a strong disincentive against actually ever curing anything since it cannot make a profit from healthy people. Many of the tests and trials for new drugs are also funded by the very same companies trying to market those drugs.

        Fortunately for science, the *real* truth usually rears it's [sic] head in the end.

        Sure, after the person suggesting that all is not as it appears to be is laughed at, ridiculed, cursed, given the old standby of "I doubt you know more than the other thousands of real scientists, mmmkay?" for daring to question the holy sacred authority of the Scientific Establishment and daring to suggest that it could ever steer us wrong or that this, too is unworthy of 100% blind faith or that it may have the same problems that plague other large institutions. The rest of us who have been willing to entertain less mainstream, more "fringe" theories that are easy to demagogue by people who have never investigated them already knew that the whole endeavor is pretty good but not nearly as good as it is made out to be by people who really want to believe in it.

        • by tgibbs (83782) on Sunday January 02, 2011 @07:34PM (#34740142)

          The pharmaceutical industry is easily one of the most corrupt industries known to man. Perhaps some defense contractors are worse, but if so, then just barely. It's got just the right combination of billions of dollars at play, strong dependency on the part of many of its customers, a basis on intellectual property, financial leverage over most of the rest of the medical industry, and a strong disincentive against actually ever curing anything since it cannot make a profit from healthy people.

          One tends to hear this sort of thing from people who don't know anything about the pharmaceutical industry, and of course this attitude is pushed very hard by people who are hawking quack cures of one sort or another, and who are thus competitors of the pharmaceutical industry.

          I'm an academic pharmacologist, but I've met a lot of the people involved in industrial drug discovery, and trained more than a few of them. People tend to go into pharmacology because they are interested in curing disease and alleviating suffering. Many of them were motivated to enter the area by formative experiences with family members or other loved ones suffering from disease. They don't lose this motivation because they happen to become employed by a pharmaceutical company--indeed, many enter industry because it is there that they have the greatest opportunity to be directly involved in developing treatments that will actually cure people.

          It is certainly true that pharmaceutical companies are businesses, and their decisions regarding how much to spend on treatments for different illnesses are strongly influenced by the potential profits. A potential treatment for a widespread chronic disease can certainly justify a larger investment than a one-time cure. But it can also be very profitable to be the only company with a cure for a serious disease. And it would be very bad to spend a lot of money developing a symptomatic treatment only to have somebody else find a cure. So a company passes up an opportunity for a cure at its peril. There is definitely a great deal of research going on in industry on potential cures.

          The real reason why cures are rare is that curing disease is hard. Biology is complicated, and even where the cause is well understood, a cure can be hard to implement. For example, we understand in principle how many genetic diseases can be cured, but nobody in industry or academia knows how to reliably and safely edit the genes of a living person in practice. It is worth noting that the classic "folk" treatments for disease, including virtually all of the classic herbal treatments that have been found to actually be effective--aspirin, digitalis, ma huang, etc--are not cures; they are symptomatic treatments. Antibiotics were a major breakthrough in the curing of bacterial diseases, but they were not created from scratch, but by co-opting biological antibacterial weapons that were the product of millions of years of evolution. Unfortunately, for many diseases we are not lucky enough to find that evolution has already done the hardest part the research for us.

    • Already debunked (Score:5, Insightful)

      by mangu (126918) on Sunday January 02, 2011 @12:52PM (#34737836)

      Is it possible that there has always been error, but it is just more noticeable now given that reporting is more accurate?

      Precisely. As mentioned in a Scientific American [scientificamerican.com] blog:

      "The difficulties Lehrer describes do not signal a failing of the scientific method, but a triumph: our knowledge is so good that new discoveries are increasingly hard to make, indicating that scientists really are converging on some objective truth."

      • by toppavak (943659) on Sunday January 02, 2011 @01:17PM (#34738014)
        Scale is also an important factor. With better statistical methodology, more rigorous epidemiology and a growing usage of bio-statisticians in the interpretation of results, we're seeing that weak associations that were once considered significant cannot be replicated in larger experiments with more subjects, more quantitative and accurate measurements. Unlike many, many other fields (particularly theology) when scientific theories are overturned, it is a success of the methodology itself.

        That's not to say that individual scientists don't sometimes dislike the outcome and ultimately attempt to ignore and/or discredit the counter-evidence, but in the long run this can never work since hard data cannot be hand-waved away forever.
        • Consider also that a result being significant to 95% confidence simply means that you would expect that same result 5% of the time purely by chance.

          But I suspect the larger problem stems from the career aspect of modern science. Sometimes the failure of a promising experiment can set you back by months or even years, not to mention clouding the horizon for future funding. It's not surprising that some will do whatever is needed to present their results in a positive light, even if that crosses ethical lines

          • by Skippy_kangaroo (850507) on Sunday January 02, 2011 @03:27PM (#34738958)

            Consider also that most researchers run more than 20 regressions when testing their data. That means that the 95% significance level is grossly overstated.

            The key is that the 95% level applies only if you don't data snoop beforehand and only if you run the regression once and only once. The true significance levels of many studies that claim a 95% level is likely to be 50% when you consider all the pretesting and data snooping that goes on in reality - not the rather idealised setup that is reported in the journal publication.

  • It's simple. (Score:5, Interesting)

    by Lord Kano (13027) on Sunday January 02, 2011 @12:31PM (#34737672) Homepage Journal

    Even in academia, there's an establishment and people who are powerful within that establishment are rarely challenged. A new upstart in the field will be summarily ignored and dismissed for having the arrogance to challenge someone who's widely respected. Even if that respected figure is incorrect, many people will just go along to keep their careers moving forward.

    LK

    • Re:It's simple. (Score:4, Informative)

      by Anonymous Coward on Sunday January 02, 2011 @01:00PM (#34737906)

      Having worked in multiple academic establishments, I have never seen that. I have seen people argue their point, and respected figures get their way otherwise (offices, positions, work hours, vacation). But when it came to papers, no one was sitting around rejecting papers because it conflicted with a "respected figure." Oftentimes, staff would have disagreements that would sometimes be an agreement to disagree because of lack of data. Is this your personal experience? Because it I don't disagree that this may occur some places, I just haven't seen it. But I want to be sure you have, and are not just spreading an urban legend.

      • by Lord Kano (13027)

        Because it I don't disagree that this may occur some places, I just haven't seen it. But I want to be sure you have, and are not just spreading an urban legend.

        That's a fair question. I have not experienced it first hand, but I have seen it as an outside observer.

        LK

      • by dr2chase (653338)
        Add to this -- reviews for conferences (in my field) are often blind -- no idea who the author is, and we're perfectly willing to admit a paper (that's done well) that might start an argument.
      • Re:It's simple. (Score:4, Interesting)

        by FourthAge (1377519) on Sunday January 02, 2011 @02:07PM (#34738410) Journal

        Oh, it happens. And if you're in the academic business, then I'm very surprised you've not noticed it.

        Politics is very important in the business of accepting and rejecting papers. It's micro-politics, i.e. office politics. It's very important to get things accepted, but in order to do so, you have to be aware of the relevant political issues within the committee that will accept or reject your work. It's hard to write a paper that doesn't step on any toes, so you have to be sure you pick the right toes to step on.

        When I was part of this business I was aware of a few long-standing feuds between academics; their research students and coworkers all took sides and rejected work from the other side. It was bizarre. It would have been funny if it had not been so pathetic. Even now I cannot watch an old Newman and Baddiel sketch [youtube.com] without being reminded of childish feuding professors from real life.

        I don't think every sort of science is like this. Probably in physics and chemistry, you can get unpopular work published just by being overwhelmingly right. But in softer non-falsifiable sciences, it's mostly about politics, and saying the right things. There are a whole bunch of suspect sciences that I could list, but I know that some of them would earn me an instant troll mod (ah, politics again!), so I'll leave it at that.

  • by girlintraining (1395911) on Sunday January 02, 2011 @12:33PM (#34737688)

    After years of speculation, the a study has revealed that scientists are, in fact, human. The poor wages, long hours, and relative obscurity that most scientists dwell in has apparently caused widespread errors, making them almost pathetically human and just like every other working schmuck out there. Every major news organization south of the mason-dixon line in the United States and many religious organizations took this to mean that faith is better, as it is better suited to slavery, long hours, and no recognition than science, a relatively new kind of faith that has only recently received any recognition. In other news, the TSA banned popcorn from flights on fears that the strong smell could cause rioting from hungry and naked passengers who cannot be fed, go to the bathroom, or leave their seats for the duration of the flight for safety reasons....

    • by onionman (975962) on Sunday January 02, 2011 @12:50PM (#34737816)

      After years of speculation, the a study has revealed that scientists are, in fact, human. The poor wages, long hours, and relative obscurity that most scientists dwell in has apparently caused widespread errors, making them almost pathetically human and just like every other working schmuck out there...

      I'll add another cause to the list. The "publish or perish" mentality encourages researchers to rush work to print often before they are sure of it themselves. The annual review and tenure process at most mid-level research universities rewards a long list of marginal publications much more than a single good publication.

      Personally, I feel that many researchers publish far too many papers with each one being an epsilon improvement on the previous. I would rather they wait and produce one good well-written paper rather than a string of ten sequential papers. In fact, I find that the sequential approach yields nearly unreadable papers after the second or third one because they assume everything that is in the previous papers. Of course, I was guilty of that myself because if you wait to produce a single good paper, then you'll lose your job or get denied tenure or promotion. So, I'm just complaining without being able to offer a good solution.

  • race to the bottom (Score:4, Interesting)

    by toomanyhandles (809578) on Sunday January 02, 2011 @12:38PM (#34737726)
    I see this as one more planted article in mainstream press: "Science is there to mislead you, listen to fake news instead". The rising tide against education and critical thinking in the USA is reminiscent of the Cultural Revolution in China. It is even more ironic that the argument "against" metrics that usefully determine validity is couched in a pseudo-analytical format itself. At this point in the USA, most folks reading (even) the New yorker have no idea what a p-value is, why these things matter, and they will just recall the headline "science is wrong". And then they wonder in Detroit why they can't make $100k a year anymore pushing the button on robot that was designed overseas by someone else- you know, overseas where engineering, science, etc are still held in high regard.
    • Don't confuse anti-intellectualism with opposition to learning - Americans still highly value practical knowledge. However, the US has always had a strong anti-intellectualism. This is nothing new. More importantly, it's a valuable cultural trait. Resistance to intellectual ideals is not always bad.

      In 250 years, the US has had two major wars on its territory. Both led to significant increases in liberty. By contrast, communism turned the 20th century into a worldwide bloodbath. The ideas pouring out of t
    • I hope you didn't read the article. If you did, your post is an example of the decline of reading comprehension in the United States. It's not an attack on science or the scientific method. If it's an attack on anything, it's how we publish scientific studies and how all too often studies are accepted as statistically significant when they are not. The article doesn't suggest that we abandon science but rather that we scrutinize it more and stop believing that the results of every study indicate the truth t

  • by water-vole (1183257) on Sunday January 02, 2011 @12:38PM (#34737730)
    I'm a scientist myself. It's quite clear from where I'm standing that to get good jobs, research grants, etc one needs plenty of published articles. Whether the conclusions of those are true or false is not something that hiring committees will delve into too much. If you are young and have a family to support, it can be tempting to take shortcuts.
    • by dachshund (300733) on Sunday January 02, 2011 @01:22PM (#34738036)

      Whether the conclusions of those are true or false is not something that hiring committees will delve into too much. If you are young and have a family to support, it can be tempting to take shortcuts.

      Yes, the incentive to publish, publish, publish leads to all kinds of problems. But more importantly, the incentives for detailed peer-reviewing and repeating others' work just aren't there. Peer-reviewing in most cases is just a drag, and while it's somewhat important for your career, nobody's going to give you Tenure on the basis of your excellent journal reviews.

      The inventives for repeating experiments are even worse. How often do top conferences/journals publish a result like "Researchers repeat non-controversial experiment, find exactly the same results"?

    • by Moof123 (1292134) on Sunday January 02, 2011 @01:32PM (#34738114)

      Agreed. Way too many papers from academia are ZERO value added. Most are a response to "publish or perish" realities.

      Cases in point: One of my less favorite profs published approximately 20 papers on a single project, mostly written by his grad students. Most are redundant papers taking the most recent few months data and producing fresh statistical numbers. He became department head, then dean of engineering.

      As a design engineer I find it maddening that 95% of the journals in the areas I specialize in are:

      1. Impossible to read (academia style writing and non-standard vocabulary).

      2. Redundant. Substrate integrated waveguide papers for example are all rehashes of original waveguide work done in the 50's and 60's, but of generally lower value. Sadly the academics have botched a lot of it, and for example have "invented" "novel" waveguide to microstrip transitions that stink compared to well known techniques from 60's papers.

      3. Useless. Most, once I decipher them, end up describing a widget that sucks at the intended purpose. New and "novel" filters should actually filter, and be in some way as good or better than the current state of the art, or should not be bothered to be published.

      4. Incomplete. Many interesting papers report on results, but don't describe the techniques and methods used. So while I can see that University of Dillweed has something of interest, I can't actually utilize it.

      So as a result when I try to use the vast number of published papers and journals in my field, and in niches of my field to which I am darn near an expert, I cannot find the wheat from the chaff. Searches yield time wasting useless results, many of which require laborious decyphering before I can figure that they are stupid or incomplete. Maybe only 10% of the time does a day long literature search yield something of utility. Ugh.

      • The sad thing is that this flow of crap has easily identified causes, but no easily identified remedies.

        --------

        Happy New Year and Good Riddance Old Year
    • That doesn't match my experience. The currency by which scientists are measured is not publications, but citations of your publications. You can publish a hundred worthless articles in obscure journals that no one ever cites, and you'll get very little credit for them. A handful of good quality, widely cited articles will do more to advance your career.
  • by IICV (652597) on Sunday January 02, 2011 @12:40PM (#34737738)

    This article has already been taken apart by P.Z. Myers in a blog post [scienceblogs.com] on Pharyngula. Here's his conclusion:

    But those last few sentences, where Lehrer dribbles off into a delusion of subjectivity and essentially throws up his hands and surrenders himself to ignorance, is unjustifiable. Early in any scientific career, one should learn a couple of general rules: science is never about absolute certainty, and the absence of black & white binary results is not evidence against it; you don't get to choose what you want to believe, but instead only accept provisionally a result; and when you've got a positive result, the proper response is not to claim that you've proved something, but instead to focus more tightly, scrutinize more strictly, and test, test, test ever more deeply. It's unfortunate that Lehrer has tainted his story with all that unwarranted breast-beating, because as a summary of why science can be hard to do, and of the institutional flaws in doing science, it's quite good.

    Basically, it's not like anyone's surprised at this.

    • by damburger (981828)
      I had already read the article having found it through PZ Myers. A lot of people like to rip on the scientific method, but few of them consider how slight the chance is that they or anyone else can successfully second-guess it.
  • by Pecisk (688001) on Sunday January 02, 2011 @12:42PM (#34737750)

    NYT article is well written and informative. It's clearly not assuming that there is something wrong with scientific method, but just asks - could it be? There is excellent reply by George Musser at "Scientific American" http://cot.ag/hWqKo2 [cot.ag]

    This is what I call interesting and engaging public discussion and journalism.

  • by rennerik (1256370) on Sunday January 02, 2011 @12:47PM (#34737784)
    > 'Which results should we believe?'

    What a ridiculous question. How about the results that are replicated, accurately, time and time again, and not ones that aren't based off of scientific theory, or failed attempts at scientific theory?
  • Bogus article (Score:5, Interesting)

    by Anonymous Coward on Sunday January 02, 2011 @12:48PM (#34737794)

    That article is as flawed as the supposed errors it reports on. The author just "discovered" that biases exist in human cognition? The "effect" he describes is quite well understood, and is the very reason behind the controls in place in science. This is why we don't, in science, just accept the first study published, why scientific consensus is slow to emerge. Scientists understand that. It's journalists who jump on the first study describing a certain effect, and who lack the honesty to review it in the light of further evidence, not scientists.

  • Publish or Perish (Score:4, Insightful)

    by 0101000001001010 (466440) on Sunday January 02, 2011 @12:56PM (#34737874)

    This is the natural outcome of 'publish or perish.' If keeping your job depends almost solely on getting 'results' published, you will find those results.

    Discovery is more prestigious than replication. I don't see how to fix that.

  • by burnin1965 (535071) on Sunday January 02, 2011 @01:03PM (#34737924) Homepage

    According to John Ioannidis, author of Why Most Published Research Findings Are False, the main problem is that too many researchers engage in what he calls 'significance chasing,' or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. 'The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,'

    Before you can question the scientific method through experimentation you first must understand and utilize the scientific process. That last quote is a massive clue that the issue is that they are stepping away from the scientific process and trying to force an answer.

    I'll go read the article but before I do I'll just note that in working in semiconductor manufacturing and development both the scientific process and statistical significance are at the core of resolving problems, maintaining repeatable manufacturing and developing new processes and products. And from my 20 years of experience the scientific process worked just fine and when results were not reproducible then you had more work to do but you didn't decide that science no longer worked and that the answer simply changed.

    I can guarantee that if we throw away the scientific process and no longer rely of peer review and replication then all those fun little gadgets everyone enjoys these days will become a thing of the past and we'll enter into the second dark age.

  • by bcrowell (177657) on Sunday January 02, 2011 @01:10PM (#34737964) Homepage

    The article can be viewed on a single page here: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all [newyorker.com]

    Not surprisingly, most of the posts so far show no signs of having actually RTFA.

    Lehrer goes through all kinds of logical contortions to try to explain something that is fundamentally pretty simple: it's publication bias plus regression to themean. He dismisses publication bias and regression to the mean as being unable to explain cases where the level of statistical significance was extremely high. Let's take the example of a published experiment where the level of statistical significance is so high that the result only had one chance in a million of occurring due to chance. One in a million is 4.9 sigma. There are two problems that you will see in virtually all experiments: (1) people always underestimate their random errors, and (2) people always miss sources of systematic error.

    It's *extremely* common for people to underestimate their random errors by a factor of 2. That means the the 4.9-sigma result is only a 2.45-sigma result. But 2.45-sigma results happen about 1.4% of the time. That means that if 71 people do experiments, typically one of them will result in a 2.45-sigma confidence level. That person then underestimates his random errors by a factor of 2, and publishes it as a result that could only have happened one time in a million by pure chance.

    Missing a systematic error does pretty much the same thing.

    Lehrer cites an example of an ESP experiment by Rhine in which a certain subject did far better than chance at first, and later didn't do as well. Possibly this is just underestimation of errors, publication bias, and regression to the mean. There is also good evidence that a lot of Rhine's published work on ESP was tainted by his assistants' cheating: http://en.wikipedia.org/wiki/Joseph_Banks_Rhine#Criticism [wikipedia.org]

  • by fermion (181285) on Sunday January 02, 2011 @01:11PM (#34737968) Homepage Journal
    The scientific method derives from Galileo. He constructed apparatus and made observations that any trained academician and craftsperson of his day could have made, but they did not because it was not the custom. He built inclined planes, lenses, and recorded what he say. From this he made models that included predictions. Over time those predictions were verified by other such as Newton, and the models became more mathematically complex. The math used is rigorous.

    Now science uses different math, and the results are expressed differently, even probabilistically. But in real science those probabilities are not what most think as probability. In a scanning tunneling microscope, for instance, works by the probability that a particle can jump an air gap. Though this is probabilistic, It is well understood so allows us to map atoms. There is minimal uncertainty in the outcome of the experiment.

    The research talked about in the article may or may not be science. First, anything having to do with human systems is going to be based on statistics. We cannot isolate human systems in a lab. The statistics used is very hard. From discussions with people in the field, I believe it is every bit as hard as the math used for quantum mechanics. The difference is that much of the math is codified in computer applications and researchers do not necessarily understand everything the computer is doing. In effect, everyone is used the same model to build results, but may not know if the model is valid. It is like using a constant acceleration model for which a case where there is a jerk. The results will be not quite right. However, if everyone uses the faulty model, the results will be reproducible.

    Second, the article talks about the drug dealers. The drug dealers are like the catholic church of Galileo's time. The purpose is not to do science, but to keep power and sell product. Science serves a process to develop product and minimize legal liability, not explore the nature of the universe. As such, calling what any pharmaceutical does as the 'scientific method' is at best misguided.

    The scientific method works. The scientific method may not be comopletey applicable to fields of studies that try to find things that often, but not, always, work in a particular. The scientific method is also not resistant to group illusion. This was the basis of 'The Structure of Scientific Revolution'. The issue here, if there is one, is the lack of education about the scientific method that tends to make people give individual results more credence than is rational, or that is some sort of magic.

  • by grandpa-geek (981017) on Sunday January 02, 2011 @03:17PM (#34738892)

    First, science has always had a political aspect. Publication reviewers are always biased by conventional wisdom among their scientific peers, and they will become critical of any submitted paper that strays from that view. A lot of careers are based on following the conventional wisdom, and threats to those careers are met with political responses.

    Second, the quest for statistical significance is based on serious misunderstanding of statistics among scientists. It has been so for decades. Publication editors are thoroughly ignorant of statistics if they demand statistical significance at the .95 or .99 levels as a condition of acceptance.

    Results that are statistically significant may or may not be clinically significant. Both factors must be considered.

    Significance levels are based on one model of statistical inference. There are other models, although those have been subjected to politics within the mathematical/statistical community. Although Bayesian statistics are now accepted (and form a critical basis in theories of signal processing, radar, and other technologies) they were rejected by the statistical community for many years. The rejection was almost completely political, because the concepts challenged the conventional wisdom.

    The basic scientific method is not a problem. The major problem is the factors in publication acceptance and the related biases and pressures to adhere to the conventional wisdom. Rejection of papers based on politics or on ignorance of statistical methods is outside the scientific method and needs to be rooted out.

  • by bradbury (33372) <Robert...Bradbury@@@gmail...com> on Sunday January 02, 2011 @03:52PM (#34739114) Homepage

    Anyone familiar with the concept of "reality on demand" knows that it is constructed on a need-by-need basis by the Blue People. Now the Blue People are not 100% reliable. Sometimes they forget to put back key pieces of reality. This is the source of the problem of failure to reproduce results. The reproduction is being attempted in a reality which is simply too different.

    Two possible solutions come to mind. Only conduct experiments where one never has to leave the room. Or maybe find a good lawyer who can negotiate a higher quality level contract with the Blue People.

If I'd known computer science was going to be like this, I'd never have given up being a rock 'n' roll star. -- G. Hirst

Working...