Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Education Medicine Stats Science

Poor Scientific Research Is Disproportionately Rewarded (economist.com) 81

Posted by EditorDavid from the granted-for-taking dept.
A new study calculates a low probability that real effects are actually being detected in psychology, neuroscience and medicine research paper -- and then explains why. Slashdot reader ananyo writes: The average statistical power of papers culled from 44 reviews published between 1960 and 2011 was about 24%. The authors built an evolutionary computer model to suggest why and show that poor methods that get "results" will inevitably prosper. They also show that replication efforts cannot stop the degradation of the scientific record as long as science continues to reward the volume of a researcher's publications -- rather than their quality.
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."
This discussion has been archived. No new comments can be posted.

Poor Scientific Research Is Disproportionately Rewarded More

Poor Scientific Research Is Disproportionately Rewarded

Comments Filter:

  • In other news... (Score:1)

    by Anonymous Coward

    Poorly written Economist articles are Disproportionately Rewarded with attention and discussion.

  • Couldn't have anything to do with short term outlook by poor management in companies? Instant results under pressure to perform on the bottom line.

    • Re: management (Score:3, Insightful)

      by Anonymous Coward

      No, a lot of this work is done in academic institutions that rely on grants to fund research. The funding cycles tend to be about three years long, and there's pressure to generate lots of publications rather than do good work. Institutions also tend to skim a lot of money off the top through F&A costs, and there's a lot of corruption involved. As a result, money isn't spent well and there's not enough to go around.

      • In medical research when we are comparing groups it is normal to specify the power/ do a power calculation

        power is a measure of the risk of finding a result when none exists (falsely rejecting the null hypothesis)

        the null hypothesis is that your two treatments are equal

        more here:
        http://powerandsamplesize.com/... [powerandsamplesize.com]

        • You are also supposed to mention those things with your bog standard psychology papers--and in several places, including the abstract, where you cover your sample size and your alpha as well as what you got as an effect size. This doesn't really do much, though, if the entire system is skewed to encourage generally weak work.

          With the neuroscience papers involved in this analysis, though, I would want to know what they're looking at. Some papers oughtn't be counted simply because the research relies on peop

      • there's pressure to generate lots of publications rather than do good work.

        True, but this is only part of the problem. There is also a huge problem of researchers producing trivial results (whether shoddy or not) because they are afraid to ask the big questions and pursue revolutionary results. If they go long, and fail, their career may be over since nobody publishes negative results. If they fake successful results, they will draw intense scrutiny and be exposed. So they play it safe and do research that nobody cares about.

        Top tier publications like Science and Nature have

        • Top tier publications like Science and Nature have some good papers.

          Science and Nature publish exciting papers with comparatively little data. They're the place you publish things that might be groundbreaking, so they get wide exposure. After a couple of years, most of the revolutionary stuff fails to pan out, but whatever does will always cite Science as the first report.

          Serious science is mostly not revolutionary

      • Our (USA) funding system was designed under the assumption that ~30% of grant applications will receive funding. Today, that number is a little more than 5%. Science prospers when practitioners compete for excellence, rather than because there isn't enough to go around.

        • Re: management (Score:1)

          by Anonymous Coward

          The cure then is to shut down approx 85% of 'research' so the funding ratio returns to 30%.

    • Couldn't have anything to do with short term outlook by poor management in companies?

      Very few companies do any published research. This is about academia and government funded labs that seek grants, not industry.

      The "short term outlook" in companies actually improves the situation, because it puts pressure on researchers to come up with real results that can be put into products, rather than bogus research papers.

      • It really doesn't help. There's all sorts of research you can't do if you're interested in turning it into a product within 2 years.

    • Re: (Score:2)

      by Potor ( 658520 )
      I think parent is taking the modern university to be a company. Even liberal arts institutions like mine are now explicitly run that way.

      • I think parent is taking the modern university to be a company. Even liberal arts institutions like mine are now explicitly run that way.

        The vast majority of universities in the US and Europe are either publicly funded or non-profits; when they are non-profit, almost all their research funding still comes from the government.

        When publicly funded or non-profit organizations tell you that they are "run like businesses", they are lying to you. Businesses need to be run such that they make a profit from what th

    • Re: (Score:2)

      by gweihir ( 88907 )

      Ah, yes. Academia is trying to copy that "success story". In companies, at some time so much trash will have accumulated from this strategically utterly demented approach that they go down the drain or at least into a major crisis.

      • Ah, yes. Academia is trying to copy that "success story". In companies, at some time so much trash will have accumulated from this strategically utterly demented approach that they go down the drain or at least into a major crisis.

        Companies generally deliver cheaper and better products year after year. In contrast, the cost of public education grows faster than inflation while the quality is either stagnant or actually declining.

        The "utterly demented approach" is that we keep shoving more and more money int

    • Couldn't have anything to do with short term outlook by poor management in companies? Instant results under pressure to perform on the bottom line.

      Most published scientific research comes from academic institutions and public funding. Therefore, these problems have mostly to do with the way the US government awards research grants to academic researchers, and the perverse incentives that creates for public and non-profit research universities in how they hire and promote.

      Corporate researchers are generally

  • You've discovered that cheaters gain an advantage.

  • This formalizes what is already known (Score:2, Insightful)

    by Anonymous Coward

    This original study is here [royalsocie...ishing.org].

    The study presents an accurate description of how research is funded in the US (biomedical sciences in particular). I can't speak in detail about other countries, but the major issues seem to be the same in other developed nations.

    The problem is how do you decide which study to fund. You have 100 scientist asking for money but you can fund only 10 of them. So you must come with some criteria that will allow you to decide which studies are worth pursuing and of these which ones ha

    • MOD PARENT UP (Score:1)

      by Anonymous Coward

      The original research describes the problem in the area of psychology, but the problems are the same there that they are in the bio-sciences. There is a positive glut of bio- and psych- science applicants, which is the real cause. Competition is incredibly fierce, which is what you are seeing in the above problems. "Produce papers or we'll show you the door" && "Get funding or we'll show you the door" && "We'll pay you nothing anyways" are really the byproducts of high competition.

      In the

  • Clearly, peer review is working as intended in academia.

    I'd rather the government fund its own labs more and fund academic less.

    • Well, I'll start by ignoring your idiotic glib comment at the beginning.

      The government research labs have or had a great reputation. Naturally, though not enough people were getting rich, so the government decided to privatise the easy, potentially lucrative management part without exposing the pressure entities to the risk that the research might not yield anything useful.

      That didn't help of course.

      But that aside, apart from taking ill informed digs at academia (government researchers are peers to academic

      • Ill informed? I work in academia, thank you very much. You don't have take my word on it. Slashdot has had many stories on the horrendous failings of peer review in the past couple of years.

        • Ill informed?

          Then what would you call it? Firstly you say this:

          Clearly, peer review is working as intended in academia.

          Now you say this:

          Slashdot has had many stories on the horrendous failings of peer review in the past couple of years.

          So which is it? Peer review working as intended or a failing of peer review?

          I work in academia, thank you very much.

          So, as an academic, you intend peer review to reward bad research? I'm not currently in academia, though I was for a while. I can't recall ever meeting anyone

  • Also happens in CS research (Score:5, Insightful)

    by gweihir ( 88907 ) on Saturday September 24, 2016 @08:42PM (#52955593)

    I have seen quite a bit of it and know of several CS PhDs that are based on bogus results. The tragedy is that people doing their research properly will take significantly longer and have much diminished chances at an academic career. And this effect propagates: First PhD students advance on bogus results, then they become professors on fraud and finally the whole research field is broken.

    • Heh.....back in the 50s someone got a PhD writing a thesis saying that bubble-sort was the fastest sorting algorithm.

      Donald Knuth later said, "That these wouldn't have passed if I had been around."

  • Scientific Research (Score:1)

    by Anonymous Coward

    Indeed. But I think there is another angle to this that, while in some ways less obviously sloppy, is in some ways worse.

    The need to publish at a furious pace might not always result in cutting corners - indeed, a lab that has done the same thing for the last 10 years might well have refined its techniques to a high level. But in order to do this, they need to remain within rigid boundaries, always using essentially the same methods. Every paper becomes a minor variation on the same old theme. It's the on

  • There are a few well-known issues: not publishing negative results, overinflating importance of your new method, and the drive to publish like crazy discourages collaboration to some extent.

    In the first issue, most well-known conferences and journals have so many positive results to choose from (partly because of the second problem), that they just don't care about the negative results. Negative results also don't draw a lot of attention in CS, as they'd pertain to someone trying their own random idea. Som

Slashdot Top Deals

As the trials of life continue to take their toll, remember that there is always a future in Computer Maintenance. -- National Lampoon, "Deteriorata"

Close