Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Medicine Science Technology

Dozens of Recent Clinical Trials May Contain Wrong or Falsified Data, Claims Study (theguardian.com) 66

John Carlisle, a consultant anesthetist at Torbay Hospital, used statistical tools to conduct a review of thousands of papers published in leading medical journals. While a vast majority of the clinical trials he reviewed were accurate, 90 of the 5,067 published trials had underlying patterns that were unlikely to appear by chance in a credible dataset. The Guardian reports: The tool works by comparing the baseline data, such as the height, sex, weight and blood pressure of trial participants, to known distributions of these variables in a random sample of the populations. If the baseline data differs significantly from expectation, this could be a sign of errors or data tampering on the part of the researcher, since if datasets have been fabricated they are unlikely to have the right pattern of random variation. In the case of Japanese scientist, Yoshitaka Fuji, the detection of such anomalies triggered an investigation that concluded more than 100 of his papers had been entirely fabricated. The latest study identified 90 trials that had skewed baseline statistics, 43 of which with measurements that had about a one in a quadrillion probability of occurring by chance. The review includes a full list of the trials in question, allowing Carlisle's methods to be checked but also potentially exposing the authors to criticism. Previous large scale studies of erroneous results have avoided singling out authors. Relevant journal editors were informed last month, and the editors of the six anesthesiology journals named in the study said they plan to approach the authors of the trials in question, and raised the prospect of triggering in-depth investigations in cases that could not be explained.
This discussion has been archived. No new comments can be posted.

Dozens of Recent Clinical Trials May Contain Wrong or Falsified Data, Claims Study

Comments Filter:
  • claims another study

    Authors of the first study were promptly sacked

    • I have been out of academia since the very early 1990s. My publications, those that went on to peer review and journal publications, are not nearly as numerous as the guy listed. No, he's a whole order of magnitude more prolific than I.

      Which makes me kinda giggle. How the hell does he even have that many publications?!?

      • How the hell does he even have that many publications?!?

        Probably doing a study on the effects of amphetamines.

    • by Paul Fernhout ( 109597 ) on Tuesday June 06, 2017 @11:12AM (#54560077) Homepage

      http://pdfernhout.net/to-james... [pdfernhout.net] "The problems I've discussed are not limited to psychiatry, although they reach their most florid form there. Similar conflicts of interest and biases exist in virtually every field of medicine, particularly those that rely heavily on drugs or devices. It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine. (Marcia Angell)"

  • Thanks for that! (Score:5, Insightful)

    by ls671 ( 1122017 ) on Monday June 05, 2017 @11:55PM (#54556965) Homepage

    Thanks for that! Now I can use that tool to generate data for my upcoming fabricated studies.

    • Well, JBS Haldane showed this technique for exposing fraud in 1939, so it's not revealing anything smart fraudsters wouldn't already know. A lot of the anomalies (though not necessarily all) are probably down to carelessness rather than fraud.

    • LOL If you're gonna put that much effort into it, you might just as well do the damned study.

      • Using a tool to generate fake test subject stats is not a lot of effort. Therefore it must be innovation. A computer saving human labor. Something to be encouraged.
  • "90 of the 5,067" (Score:5, Insightful)

    by Nutria ( 679911 ) on Monday June 05, 2017 @11:56PM (#54556967)

    That's... less than 2%. Naturally, we want it to be 0%, but 1.8% is nothing to generate scare headlines over.

    • by hey! ( 33014 ) on Tuesday June 06, 2017 @12:08AM (#54556995) Homepage Journal

      You stole the words right from my mouth: 90/5067? That's significant at the p < 0.02 level!

    • by ShanghaiBill ( 739463 ) on Tuesday June 06, 2017 @12:22AM (#54557041)

      That's... less than 2%. Naturally, we want it to be 0%, but 1.8% is nothing to generate scare headlines over.

      They only caught the dumb ones. It would have been easy to generate fake data that fits a known distribution. For instance, in Python, just use numpy.random.normal instead of numpy.random.uniform.

      The 2% is just the floor. The actual fraud and/or incompetence rate is likely higher.

      • by joneil ( 677771 )

        One other thing to consider.
        I am a funeral director. I see deaths first hand from medical mistakes and malpractice. In fact, I only see that "2%".
        You know that old joke "an undertaker is somebody who cleans up the doctor's mistakes"? Well, more truth to it than you might want to know.

        So, for all you people who say "it's only 2%", you come, sit with me when I deal with a family who has had a death because they were part of that "2%". You look them in the eyes, and say" hey, science is still good

      • by AK Marc ( 707885 )
        You can't fake the expected results if you use a RNG. Also, RNGs don't generate Normal results. They generate random results. Most real-world "random" events are not "random", but are "normal". So a computer-generated RNG would fail to make reasonable results. That would be obvious to anyone who has faked a result. You need to overlay a normal distribution on your RNG. It's easier to just guess results, using your brain as both the RNG and normal distribution.

        The good frauds can't be caught. The ba
        • >You can't fake the expected results if you use a RNG
          Yes you can.

          >Also, RNGs don't generate Normal results.
          They most certainly can. Look up the Box Muller method or the Ziggurat algorithm.

          >They generate random results.
          If the designer knew what he or she was doing. Usually they don't.

          >Most real-world "random" events are not "random", but are "normal".
          That depends on how you measure it. Poisson distributions are an example of something you can force by choosing your measurement method.

          >So a com

        • Most real-world "random" events are not "random", but are "normal".

          And when you are doing a clinical trial on some potential new drug, your test population is never "random" or "normal". It is selected for the disease or condition that you are trying to fix. It will not be unusual for 100% of test population to have a BMI of 50 when you're testing a new diet drug for significantly overweight people, for example. The fact that a very large proportion of those same people will also have high blood pressure and high cholesterol is not unusual, it is to be expected. And oh, my

        • It's trivial to get an RNG to generate normally distributed results by applying the central limit theorem. Consider generating a random number between 1 and 10. Run it 100,000 times and track the results and you'll have a uniform distribution, or should unless your RNG is terrible. Instead of doing that, generate 10 random numbers and take the average (which will always be a number between 1 and 10) and record that, then repeat that 100,000 times. You end up with a normal distribution. You can write the cod
    • by Anonymous Coward

      Clickbait aside, I support investigating these 90 cases of suspected fraud. The fact that the suspicious studies are all related to medicine does not make this less urgent.
      In an ideal world all studies would be repeated by multiple independent teams for confirmation, but the remaining 4977 ones will probably be given low priority in reality.

      On a sidenote, /. seems not to like the HTML code for a non-breaking space.

    • Re:"90 of the 5,067" (Score:5, Informative)

      by SharpFang ( 651121 ) on Tuesday June 06, 2017 @03:06AM (#54557477) Homepage Journal

      Seems like artifact of randomness - Prosecutor's Fallacy [wikipedia.org].

      Yes, some will be genuine falsifications. But some WILL be genuine results.

      You write a paper on a list of 1000 tosses of a coin, noting each result. The chance for the coin to land on edge in one toss is around 1 in 100,000.

      Then your paper is reviewed along with 100,000 others. If you have the coin land on edge more than once in your dataset, it's flagged as a falsified dataset.

      Roughly 10 papers in the 100,000 tested flagged as falsified will be false positives.

      ------

      Statistical results are subject to the same randomness as single tests contributing to these results. The scale of the randomness is reduced by a factor related to the number of tests, but still exist. And take enough correctly obtained statistical results, and you WILL find outliers.

      • by Anonymous Coward

        This sort of effect is always a concern. But, per TFS, they found 43 trials in which the "measurements that had about a one in a quadrillion probability of occurring by chance". Unless there were a quadrillion trials (unlikely), the Prosecutor's Fallacy isn't relevant here.

      • Some of the ones that fail the 1 in 10,000 test are quite possibly an effect of randomness, although 82 out of 5015 is a much higher failure rate than would be expected. And 43 of those 5015 having a probability of less than 1 in 10^15 really isn't a plausible random artefact.

      • Huh... Thanks. I'd never read/seen that fallacy. In my defense, I was on the debate team at the collegiate level, but this fallacy was coined more recently than my experiences in said team. Yeah, I'm that old...

        Anyhow, I am trying to wrap my head around it.

        The odds of winning without cheating at 1:1000.
        The odds of winning with cheating are 1:100.
        They won, ergo the most probable reason for their winning was they cheated. Which, while true, doesn't actually mean that they cheated - it just means someone doesn

        • Yes. It's a good heuristic to point out suspects for more thorough tests (which may be too expensive to conduct on the whole statistic base) but it isn't a proof by itself.

        • Not quite, because knowing the odds of winning with cheating by itself says nothing. It's like having over one thousand people play, but always concluding that the winner cheated because the odds that they could win (.1%) are so small that they must have cheated, without considering that when you have that many people playing, even though the odds of winning are individually low, there are enough people to make it likely that at least one person wins.

          If you do a binomial calculation with .1% chance of su
      • Seems like artifact of randomness - Prosecutor's Fallacy [wikipedia.org].

        Yes, some will be genuine falsifications. But some WILL be genuine results.

        You write a paper on a list of 1000 tosses of a coin, noting each result. The chance for the coin to land on edge in one toss is around 1 in 100,000.

        Then your paper is reviewed along with 100,000 others. If you have the coin land on edge more than once in your dataset, it's flagged as a falsified dataset.

        Roughly 10 papers in the 100,000 tested flagged as falsified will be false positives.

        You're assuming the authors of the study weren't very good at stats.

        If their standard for false data was 2/1000 coins landing on edge then yes, they got false positives.

        If their standard was 100/1000 coins landing on their edge then I'm pretty sure those data sets were wrong.

    • by Dunbal ( 464142 ) *
      Those are the ones easily caught. There are bound to be more. How many turds do you want to see in the punchbowl before you stop wanting to drink punch?
    • It also seems like poorly constructed samples might fail this test, so it is possible that some fraction of the 43 are not intentional fraud, but just poor science. My impression is that it can be tricky to construct a good sample for a lot of medical studies, so it's not surprising that some of them are imperfect. I can also think of situations where correctly constructed samples might fail this test, but it's possible that the analysis is accounting for that.
  • Outlier data (Score:4, Interesting)

    by Vitriol+Angst ( 458300 ) on Tuesday June 06, 2017 @01:12AM (#54557199)

    So 90 of the 5,067 were outlier-like data and this is concluding results on these outliers.

    I knew that publish or perish was ruining science, but this is actually the most heartening news I've heard of its credibility.

    I've learned that less than 1.8% of these studies used non-extreme crazy data. My faith in science is restored!

  • A study should be repeated by an org that has no skin in the game per results. They should be paid to test and get the same amount of compensation regardless of outcome. A random lottery should decide the head managers/researchers for any given repeat.

  • That word strikes again. "May". There is never a time when recent studies definitely had no falsified data. The opposite of this would be far more newsworthy, which is usually a good sign that someone has been wasting their time. There always has to be skepticism of studies, that's why replication is required.
  • ...But, did he use a bonferroni correction to compute his p-values, since this is a classic data dredge? Sure, his method will turn up true positives (and did, for at least one known offender) but what remains to be seen is the false positive rate and the lawsuit rate, since skewed distributions could have many causes some of which are benign and this is pretty serious defamation of character if one casts aspersions without secondary supporting evidence of malpractice.

    In other words, are his "positives" re

  • Scott Gottlieb, the current head of the FDA, wants to end drug trials. "The free market will put the bad actors out of business."

  • Most of those studies are probably not blinded. I wouldn't be surprised if all of the flagged ones are not. There doesn't have to be any fraud at all. If you know what the groups are, your brain will introduce its own bias, without you even knowing about it.

  • I didn't read TFA, but

    90 of the 5,067 published trials had underlying patterns that were unlikely to appear by chance in a credible dataset

    The usual threshold of statistical certainty used for publishing scientific results is 95% (sometimes 98%). That is, a result becomes noteworthy enough to publish if there's a 5% or lower chance of it happening simply due to random chance.

    90 studies out of 5,067 is 1.8%. Which is below the 5% you'd expect from a 95% threshold, and even the 2% you'd expect with a 98

If it's not in the computer, it doesn't exist.

Working...