Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Bug Software Science

MRI Software Bugs Could Upend Years Of Research (theregister.co.uk) 95

An anonymous reader shares a report on The Register: A whole pile of "this is how your brain looks like" MRI-based science has been invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny "voxels". Software, rather than humans, then scans the voxels looking for clusters. When you see a claim that "scientists know when you're about to move an arm: these images prove it", they're interpreting what they're told by the statistical software. Now, boffins from Sweden and the UK have cast doubt on the quality of the science, because of problems with the statistical software: it produces way too many false positives. In this paper at PNAS, they write: "the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
This discussion has been archived. No new comments can be posted.

MRI Software Bugs Could Upend Years Of Research

Comments Filter:
  • by damn_registrars ( 1103043 ) <damn.registrars@gmail.com> on Tuesday July 05, 2016 @10:22AM (#52448283) Homepage Journal
    The research is on fMRI - the F stands for Functional. As it mentions later in the summary this is used to try to associate regions of the brain with specific functions. This is not the same as the structure of the brain itself. What we see in terms of actual brain structures - folds, regions, etc, is still very much valid. We're just not so sure about the functional assignments that we've held on to for a while now.
    • Regardless, reproductible research [reproducibleresearch.net] is desirable.
      • Regardless, reproductible research is desirable.

        I agree with you 100% on that (assuming of course that you meant reproducible and not reproductible). Honestly though there are few fields of modern science that are not having at least some reproducibility issues. Some suffer it more than others, of course, but it is a rather pervasive problem. As much as neurology is a well established medical speciality, there is a lot we still don't know about how the brain works. fMRI and other tools were supposed to help but without a solid foundation we're still

        • assuming of course that you meant reproducible and not reproductible

          Sorry, that was a typo. :)

          • assuming of course that you meant reproducible and not reproductible

            Sorry, that was a typo. :)

            I figured as much, but thought I'd check in case you are involved in (or want to recruit others to partake in) some sort of cutting-edge HVAC research.

    • by JoeMerchant ( 803320 ) on Tuesday July 05, 2016 @11:47AM (#52449125)

      Whenever I've read an fMRI "research" paper, it seems like the f should be standing for "full of ____", because the sample sizes are laughably small, the data are fuzzy and interpreted with a lot of handwaving, and the correlation between oxygen uptake and the fMRI signal itself is very weak, finally somebody has gotten around to calling BS on the whole field.

      • Dunno, it seems like it could open a whole new field. Research work on fresh salmon.

        Mmmm. Salmon.

  • by Anonymous Coward

    It's not exactly new this issue. Through the link is a study of the active regions of the brain of a dead salmon....

    http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

  • by Anonymous Coward on Tuesday July 05, 2016 @10:30AM (#52448361)

    A friend of mine as worked in the social sciences (cue /. laughter, but bear with me) and they were forced by the university to use a closed source statistical package for all their data processing. So anyway, she got some really dubious results and she preferred to do her own maths, so she did, and lo! completely different results. That was the start of a research project which concluded that the closed source package contained a rounding error that basically filtered all minorities out of the data set, which is kind of sad if you're doing research on minorities.
    People trust their software too much, are too lazy to do their own maths, don't really want to have got anything to do with data processing even though that's their job, and universities force bad software on their employees. This is an institutional problem that goes way beyond MRI research.

    • A friend of mine as worked in the social sciences (cue /. laughter, but bear with me) and they were forced by the university to use a closed source statistical package for all their data processing. So anyway, she got some really dubious results and she preferred to do her own maths, so she did, and lo! completely different results. That was the start of a research project which concluded that the closed source package contained a rounding error that basically filtered all minorities out of the data set, which is kind of sad if you're doing research on minorities. People trust their software too much, are too lazy to do their own maths, don't really want to have got anything to do with data processing even though that's their job, and universities force bad software on their employees. This is an institutional problem that goes way beyond MRI research.

      I had a university level Statistics "professor" once tell me that I didn't need to know how my calculator created a box plot, etc etc because I could just use someone else's statistics library instead of writing my own. While in general I agree that there is no point in reinventing the wheel, I felt like I ought to learn how such things work.

      • I had a university level Statistics "professor" once tell me that I didn't need to know how my calculator created a box plot, etc etc because I could just use someone else's statistics library instead of writing my own. While in general I agree that there is no point in reinventing the wheel, I felt like I ought to learn how such things work.

        I do a *ton* of statistical work in my day job, and if I were to write a book or teach a class, I would recommend two things:

        1) Always look at the data
        2) Always write your own functions

        The reason for this has to do with the basic nature of statistics. If you make a mistake in normal software, the error is usually patently visible or benign. Often times the software works fine and does its job and the results are correct, even if it has bugs.

        In statistics however, if you make a mistake the results get closer

      • I had a university level Statistics "professor"

        What other levels can a professor have?

    • Worth mentioning that not long ago, someone got fMRI results from dead salmon [wired.com].
    • Problem with closed source and science.

      Similarly there was a court case in Florida where people were suspicious of Breathalyzer results. Police use one produced by a company with closed source code. Court ordered them to open it up for inspection. They tried the "Trade Secrets" argument and refused. Court disagreed and starting fining them every day until they release the code. Once they did it was found to be horrible, and inaccurate, invalidating thousands of court cases... As it turned out they knew it w

  • by Anonymous Coward

    It's a matter of time before this happens with global warming, too. It's well known that the temperature record is adjusted, supposedly to remove biases. However, if you look at the unadjusted data, it fits the solar cycle perfectly, with temperatures declining over the past few decades, coinciding with solar dimming. The adjustment looks like a hockey stick, though, which can explain the entirety of the supposed warming. The National Climatic Data Center once had these figures on their website, though they

    • by Anonymous Coward

      And gravity. Everyone keeps using our type of matter in their experiments, where the inertial mass and the gravitational mass of everything is nearly identical if you use open source software to make the statistical comparisons! But these two masses are only the "same" if you use mathematics which can be reviewed for accuracy. If you use the correct proprietary software (you have to preserve the trade secrets), you'll see the two masses are different (because of ghosts). That's why only properly equipped sc

    • by TapeCutter ( 624760 ) on Tuesday July 05, 2016 @06:25PM (#52452859) Journal

      It's a matter of time before this happens with global warming, too.

      Well financed "skeptics" have been busting a gut for over 20yrs trying to prove your conspiracy theory, they have done nothing but bring the word "skeptic" into disrepute.

  • by daenris ( 892027 ) on Tuesday July 05, 2016 @10:48AM (#52448493)
    The paper has been available as a preprint for awhile now, and my lab has discussed it internally and I've also paid attention to outside coverage. The key issue that the paper reports is that false positive rates are two high for most existing software WHEN using a specific type of test under a specific set of conditions. They show that voxelwise familywise error (FWE) correction actually seems to work reasonably or even conservatively. Cluster level FWE correction (looking for groups of voxels that are active) fails when using a very liberal cluster-defining threshold, but works reasonably well when using a more stringent cluster defining threshold. It also says nothing about the performance of another very common correction method that is frequently used in fMRI studies (false discovery rate or FDR).

    I'm not really sure how extensive the group of findings that these issues actually affect is, but it's certainly not 40,000 as is claimed in the paper's significance section. Many of the earlier papers (and even more recent) likely used uncorrected statistical tests, so are suspect for entirely different reasons from this issue. Of the ones that use correction, the findings in this paper only call into question the results for those that are using FWE cluster correction with a cluster defining threshold that is too liberal (likely > 0.001, the paper's findings suggest that at 0.001 the familywise error rate is in the ballpark of the desired 5%). Those using a cluster defining threshold of p=0.001 or lower are likely fine, and those using a different correction method like FDR are unknown as to my knowledge there isn't currently any similar paper on that correction method.

    You can also check out this technical report by some other big names in imaging that basically says that this result is known and expected for overly liberal cluster defining thresholds:
    http://www.fil.ion.ucl.ac.uk/s... [ucl.ac.uk]
    • by BenBoy ( 615230 )
      Please keep your "facts" out of my outraged 'science is soooooo stupid' thread ...
    • We've also been looking this over. It doesn't exactly invalidate previous studies that used high clustering threshold of p0.05, it just indicates that they are not as robust as once thought. The paper itself could change what reviewers accept though. Maybe some reviewers will say that based on this paper, only analyses using a FLAME1 or permutations method should be accepted. Much like registering EPIs directly to the standard template is frowned upon. It depends on the reviewer and the justification for yo

    • Thank you, comments like yours are the reason I still come here.
    • by ceoyoyo ( 59147 )

      There have been a few other papers criticising FWE clustering lately. It's always struck me as kind of an iffy concept. Even the simpler non-clustering techniques, although they seem to do more or less what they advertise, really should be regarded as exploratory and checked by proper hypothesis driven replication studies.

  • i've been wanting to learn R and thought about doing some maths on the raw data and compare it with the released results. mostly looking at trends at specific weather stations compared to official numbers
  • by medv4380 ( 1604309 ) on Tuesday July 05, 2016 @11:10AM (#52448709)

    The researchers used published fMRI results, and along the way they swipe the fMRI community for their “lamentable archiving and data-sharing practices” that prevent most of the discipline's body of work being re-analysed.

    So the raw data isn't being saved so that someone else can independently verify the results. No checking the computers math, no checking the researchers settings on the machine. Just blanket trust for the people and the machine, and purging of any way of poking holes in someones findings. Even if this wasn't caused by a software bug the lack of archiving the raw dataset so that it can be rerun when software improvements are made is just infuriating.

    • by guruevi ( 827432 ) on Tuesday July 05, 2016 @11:45AM (#52449091)

      That is because MRI data (at least in the US) is protected by HIPAA. You can reconstruct enough identifiable features from raw data plus you have to record quite a number of other features (age, weight etc. for radiation calculations) that almost all MRI data falls under HIPAA when it comes to redistributing the raw data. If you strip all that out (skull stripping, DICOM anonymize), it's no longer raw data AND it becomes very hard to distinguish things like image orientation.

      • You can reconstruct enough identifiable features from raw data plus you have to record quite a number of other features (age, weight etc. for radiation calculations)

        There's no ionizing radiation in an MRI. The age is not needed for the scan either. The weight is needed to calculate the SAR (specific absorbtion rate). In simple terms, it's so you don't cook the patient since RF pulses are being used to disrupt the magnetic field. These heat up the patient.

        . If you strip all that out (skull stripping, DICOM anonymize), it's no longer raw data AND it becomes very hard to distinguish things like image orientation.

        Only data that make it possible to identify the patient. The vast majority of the DICOM header does not. The patient name, MRN, etc. must be removed. The image orientation, flip angle, TR, FOV, slice thick

        • by guruevi ( 827432 )

          That is what I meant, RF radiation. I work in MRI, no vendor I've seen uses anything standard except for open source software. Siemens doesn't even include a lot of tags so orientation after stripping requires physical markers.

          • Then perhaps they should consider adopting a standard that can adhere to HIPPA privacy rules, and provide a way to re-verify the analysis. Otherwise the research half of the fMRI scans are utilizing HIPAA as a shield to protect their conclusions. I work in Study Research that has to adhere to HIPAA rules, and there is quite a bit that can be included in a dataset sanitized of identity information. Otherwise no one would have their study retracted due to fraud because they could hide the dataset from scrutin
          • by ceoyoyo ( 59147 )

            Not sure what you're doing, but you're doing it wrong. The DICOM standard includes very specific tags for identifying orientation unambiguously. In hundreds of thousands of images over a decade and a half I've never seen a DICOM file from an image acquisition system that didn't properly implement them.

            http://dicom.nema.org/medical/... [nema.org]

            Also, if you can't figure out all the directions except L/R with the skull stripped, you should probably take an anatomy class. Or look at a scan.

    • The whole field is full of "too expensive to do good science, but let's publish anyway." Magnet time runs $500/hr, too expensive to get an adequate number of subjects, or trials with a given subject, fMRI data is a time series of complex volumes - up until recently it was "too expensive" to store 1-2GB of data per subject-trial, but, but, it's just so cool, we wanted to share (and get our name on a publication.)

    • There are two parts to this:

      1) The raw data may or may not be saved. But it costs money to save the data. Once the research study is finished, the money is gone too, so there may be no way to pay for storage to save the data. Some researchers may hold on to it, some delete it. Until very very recently, there was no universal funded repository for neuroimaging data either. Now the NIH mandates, and pays for, the long term archiving of all NIMH funded imaging studies, including genetics.

      2) The other problem i

  • Meh! I already those studies (video game make you a psychopath/serial killer etc.) were crap, with an agenda.
  • When a story embeds the same link three times in a row (once in the mast, then twice in the article text) pretty please with sugar on top display the redundant links with "[register.com]" following the link, just like it does in my configured article view.

    Or, clever idea, you could display "[repeat link]" in each case where a link is repeated.

    If you're feeling extra ambitious—but you don't wish to interrupt your feverish efforts to deliver proper Unicode support one minute more than absolutely necessa

  • by ADRA ( 37398 )

    I love it when people run studies to actually verify / build-upon previous results. What I'm really seeing from this article is that there's a lot more "plug numbers into tool" research going on than I first expected. I would've hoped that the tools themselves would output confidence coefficients so that at least the researchers would have a clue as to how much magic they'd come up with...

  • One famous example of error related problems with fMRIs is the infamous brain scan of the dead salmon. I'm not sure if I can post a link but its: http://www.wired.com/images_bl... [wired.com]
  • Scanning Dead Salmon in fMRI Machine Highlights Risk of Red Herrings [wired.com]

    Neuroscientist Craig Bennett purchased a whole Atlantic salmon, took it to a lab at Dartmouth, and put it into an fMRI machine used to study the brain. The beautiful fish was to be the lab's test object as they worked out some new methods.

    So, as the fish sat in the scanner, they showed it "a series of photographs depicting human individuals in social situations." To maintain the rigor of the protocol (and perhaps because it was hilarious)

    • by gweihir ( 88907 )

      Glad to see that there is at least one actual scientist in that field. The others seem to be mainly morons with big mouths.

  • It will probably only take 20 years for them to come to the same conclusion about the detection of gravity waves

  • But sure enough, they found nothing.

  • by gweihir ( 88907 ) on Wednesday July 06, 2016 @12:26AM (#52454361)

    I.e. people seeing what they expecting to see, not what is there. With the huge egos, (but not nearly as large skills) in people doing Neuro-"Science" these days, I am entirely unsurprised. The grand claims about what they know and how things work have been a dead giveaway for years. Things are not that simple in practice.

"Live or die, I'll make a million." -- Reebus Kneebus, before his jump to the center of the earth, Firesign Theater

Working...