Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
AI Medicine

AI As Good As Doctors At Checking X-Rays 83

Posted by BeauHD from the future-of-medicine dept.
A new study from the University of Warwick found that artificial intelligence can analyze X-rays and diagnose medical issues better than doctors. The BBC reports: Software was trained using chest X-rays from more than 1.5m patients, and scanned for 37 possible conditions. It was just as accurate or more accurate than doctors' analysis at the time the image was taken for 35 out of 37 conditions, the University of Warwick said. The AI could reduce doctors' workload and delays in diagnosis, and offer radiologists the "ultimate second opinion," researchers added. The software understood that some abnormalities for which it scanned were more serious than others, and could flag the most urgent to medics, the university said.

To check the results were accurate, more than 1,400 X-rays analysed by the software were cross-examined by senior radiologists. They then compared the diagnoses made by the AI with those made by radiologists at the time. The software, called X-Raydar, removed human error and bias, said lead author, Dr Giovanni Montana, Professor of Data Science at Warwick University. "If a patient is referred for an X-ray with a heart problem, doctors will inevitably focus on the heart over the lungs," he said. "This is totally understandable but runs the risk of undetected problems in other areas".
This discussion has been archived. No new comments can be posted.

AI As Good As Doctors At Checking X-Rays More

AI As Good As Doctors At Checking X-Rays

Comments Filter:
  • ... damning with faint praise.
    • They've got it backwards. The doctors are as bad as the AI. I suspect with improvements in the dataset along with a mound of other patient data that an AI might be able to sift through that they'll become better in time, but what we have now is more of a best-we-can-do than anything.

      • I don't understand. I read the article, and it says nothing about the general success rate. What makes you conclude that the AI is bad at it?

  • not surprised (Score:4)

    by nicubunu ( 242346 ) on Tuesday December 12, 2023 @10:56AM (#64076009) Homepage

    I am not surprised, as I show the same X-ray to 3 doctors, each of them will say something different.

    • I am not surprised, as I show the same X-ray to 3 doctors, each of them will say something different.

      Perhaps. But if two say the growth is benign, and one says it's cancer, and then you die six months later, obviously only one of them was right.

      The goal of AI radiology isn't to match what a human radiologist would diagnose but to make a correct diagnosis.

      So they are trained on images where we know the correct diagnosis through retrospective analysis of outcomes.

      AI radiology is already good enough that the only reason humans are still in the loop is for legal liability.

    • Re: (Score:2)

      by dvice ( 6309704 )

      They did this test before and some doctors rated the same X-ray differently in different days.

    • Or that according to AI and research done in the past 18 months -- the greatest determiner as to whether a photo of skin coloration is cancer or not --- the presence of a ruler on the photo. Ruler on photo, cancerous. No ruler on photo, non-cancerous.

      • Re: (Score:2)

        by narcc ( 412956 )

        We have ways to show how much different parts of an image affect the output. That should help us avoid problems like that. At least the more obvious ones.

  • AI uses (Score:5, Insightful)

    by TheStatsMan ( 1763322 ) on Tuesday December 12, 2023 @11:00AM (#64076023)

    We're slowly learning that the best use cases for AI involve a human-in-the-loop.

    Typically medical experiments compare against the standard of care. In this case, that's a doctor diagnosing a patient from an x-ray. The article is very light on the details of the experiment, but I imagine three groups: 1) doctor alone, 2) model alone, 3) doctor/AI tandem. Medical imaging interpretation is just one area where models perform extremely well (the training data is well suited to the problem). It's irresponsible to completely outsource the diagnosis to a model, but a human-in-the-loop system is clearly time-saving, cost-saving, and apparently patient saving.

    I work in a completely different field, but we are already using models in analogous scenarios, and they improve prediction compared to human-diagnoses alone. This isn't a surprising result.

    • The software, called X-Raydar, removed human error and bias, said lead author, Dr Giovanni Montana, Professor of Data Science at Warwick University.

      It clearly does not remove human error and bias at all, it retains the human error and bias of the humans used to verify and provide positive reinforcement feedback to the AI algorithm. They would be better off stating this up front because AI isn’t magic or moral or altruistic or unbiased rather it relies on training, it’s not clear a general intelligence would be either going off the general intelligence that already exists. We all saw what happened to Microsoft’s Tay.

      • Re:AI uses (Score:5, Insightful)

        by Calydor ( 739835 ) on Tuesday December 12, 2023 @11:19AM (#64076049)

        Something like this has a few additional options to weed out most of the bias.

        First of all, time. If the X-ray was taken five years ago it's very possible that any illness in the patient has been confirmed or discovered since. As such, even an at-the-time undiscovered lung problem may have shown itself by now.

        Secondly, when the AI pops out a diagnosis the doctors checking its results will look for traces of that illness specifically, as opposed to the example of a lung problem in someone being sent in for a heart problem.

        Third, these are the early days. They are figuring out HOW to make the AI do these things. Five, ten years down the line with additional processing power, focus, etc. the AI may well have a vastly better detection rate than a doctor, making it the first to look, and a doctor to confirm the results when he knows exactly what to look for.

        • In the case of Yale university, they have been experimenting with AI for years on CT & MRI scans. Similar results, over 90% accuracy detecting tumors, which is slightly better than the results from radiologists. However, I completely agree with the sentiment "confirm the results". There absolutely needs to be human oversight, but this could be a great tool for triage.

          On a random note, if I recall correctly, Yale partnered up with Toyota of all people for their AI software. That is hearsay as I'm hav

          • Re: (Score:3)

            by r0nc0 ( 566295 )
            Folks have been using statistically trained models to "read" X-ray and other scans for quite some time - in other countries. In 2015 I saw the results of one of these models and it was quite good with accuracy better than 95% at the time. It was used in many countries in Africa and elsewhere that don't have enough clinicians to provide care or read the scan/xray. The discussion with the team that built the model was that there was a much longer road to acceptance in places like the US where medical liabilit

        • First of all, time. If the X-ray was taken five years ago it's very possible that any illness in the patient has been confirmed or discovered since. As such, even an at-the-time undiscovered lung problem may have shown itself by now.

          But the bias remains because many of these ailments go on to presenting yet remain undiagnosed and officially attributed [nih.gov].

          Secondly, when the AI pops out a diagnosis the doctors checking its results will look for traces of that illness specifically, as opposed to the example of a lung problem in someone being sent in for a heart problem.

          This can lead to a diagnostic cascade of incidental findings [jamanetwork.com], it would have to be shown that the net effect is positive, simply showing that a patient has some other small defect can lead to harmful outcomes instead of helpful.

          Third, these are the early days. They are figuring out HOW to make the AI do these things. Five, ten years down the line with additional processing power, focus, etc. the AI may well have a vastly better detection rate than a doctor, making it the first to look, and a doctor to confirm the results when he knows exactly what to look for.

          Sure, it could someday be made into a useful tool, but human bias is very difficult to remove when it’s just humans all the way down. Safety and efficacy

          • So? "It's not absolutely perfect, so lets stay with the current system that we know routinely misdiagnoses people and make no efforts to improve that" seems like a terrible idea.

            • So? "It's not absolutely perfect, so lets stay with the current system that we know routinely misdiagnoses people and make no efforts to improve that" seems like a terrible idea.

              Practicing evidence based medicine is more than just hopping on the current AI trend and saying it can’t be biased because it’s a machine. All I did is point out the fallacy in the statement made in TFA. You put up this strawman, it’s your construct.

    • Humans will still be allowed to override AI even if it harms overall system performance on average. The one case people cannot stand is when a machine makes a mistake that a person would not have made. To prevent that, we will overlook cases where the human made things worse by overriding system, even if that happens more often.

    • Re:AI uses (Score:5, Insightful)

      by bjdevil66 ( 583941 ) on Tuesday December 12, 2023 @11:45AM (#64076111)

      I'm 100% behind this kind of AI application. This is one of the best ways AI can and should be used - for everyone's mutual benefit.

      But..... How long before the C-level executives at insurance companies and other medical firms deem it safe (enough) to remove the doctors from the loop to increase profits for their next quarterly report and secure their next record-setting bonuses?

    • Re: (Score:2)

      by dvice ( 6309704 )

      > We're slowly learning that the best use cases for AI involve a human-in-the-loop.

      You are now rating AI based on some weak second rate AI. We can already make better than that.

      Best use case is when you don't need human at all. Within a couple of years Google's Gemini most likely can do both the inhumanly accurate part for most cases and very general human part for corner cases, e.g. to detect failed images or jokes from colleagues, all in one AI. It could be that it already can do this based on public i

    • I wonder if there will be ANY confidentiality similar to doctor/patient confidentiality.... I'm pretty sure the answer is no.

  • logical progression (Score:3)

    by Virtucon ( 127420 ) on Tuesday December 12, 2023 @11:25AM (#64076063)

    ML algorithms are a known, reliable quantity. The medical field already uses them for the analysis of EKG strips for possible heart issues. Doing this with X-ray images is a rational step with appropriate follow-up with a professional for any medical treatment.

  • An AI is going to force my illness into one of the 37 things it knows from its training data.

    If I have something less common then it will still put me in one of those 37 boxes whereas a doctor may look at my charts and say, "Oh hey this is weird, let's do more tests, I'm not sure what this result means".

    AI -can- often be good for the common cases but could kill you the the other times.

    AI is a helpful tool to humans but not a replacement for human judgment and experience.

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      "An AI is going to force my illness into one of the 37 things it knows from its training data."

      Absolutely. As opposed to one of the five things your physician is most familiar with, probably the one or two they specialise in.

      There are a lot of studies showing that physicians over diagnose conditions they see often. That's the problem with experience.

    • Re: (Score:2)

      by dvice ( 6309704 )

      You are overestimating human doctors. I went to 3 different specialists. All of them were certain that they knew what was wrong with me, and all of them were wrong. I still don't know what is wrong with me, so currently I'm just waiting for the AI to get good enough to tell me. There are plenty of cases where humans are unable to identify e.g. common decease, that happens to be very uncommon in the area where the doctor lives.

      You are also underestimating what AI can do in the future. In a couple of years, A

      • Took me twenty-eight years and five different doctors to get a diagnosis of 'anxiety' for 'why do I want to throw up in public.' This is after a battery of tests ranging from 'lets stick a camera up your ass' to 'drink this radioactive chalk and smile for the camera.'

        And this was during the prozac craze when everybody and (literally) their dog was getting anti-anxiety prescriptions.

        The solution was 'here's a pill; take it two hours before you're going to be somewhere you know will kick off the anxiety you

    • As opposed to what? At least the AI will know 37 things. And doctors kill people through lack of knowledge ALL THE TIME. The standard you would hold the AI to is actually much higher than that we hold doctors too, save through malpractice suits.

      I guarantee that almost all doctors will miss the 38th illness.

  • Virtual doctor says "You've got leprosy, goodbye."

    you will be billed $399 for this service

  • Which actual MDs basically do not do. We have seen this with IBM Watson something like 10 years ago: Usually better than an MD, but occasionally kills a patient by getting it completely wrong.

    • Re: (Score:2)

      by Zak3056 ( 69287 )

      Which actual MDs basically do not do.

      Uh huh. A medical resident who has been on shift for more than 20 hours and is in hour 100 for the current week certainly won't hallucinate due to sleep deprivation, right?

    • Re: (Score:2)

      by dvice ( 6309704 )

      Hallucination is a problem only for the traditional AIs, it will be solved in a couple of years (by 2028, 50% certainty) by Google Deepmind. At least according to their AI researcher and leader.

      And humans make very stupid mistakes also and they make them very often. Most of them just go unnoticed. I know this, because I have access to a huge database filled with human inputted information and I made a tool to check this data for errors and there are millions of errors in there. Some are made by laziness, so

      • Re: (Score:2)

        by gweihir ( 88907 )

        Hallucination is a problem only for the traditional AIs, it will be solved in a couple of years (by 2028, 50% certainty) by Google Deepmind. At least according to their AI researcher and leader.

        At this time, having observed several AI hypes, my confidence level in what AI "researchers" predict is strongly negative.

    • Sure, but that was ten years ago.

      It wasn't that long ago that it was a big deal that Deep Blue barely ecked out a victory against Kasparov in a chess tournament. Nowadays, it's widely acknowledged that a modern engine running on a desktop computer will destroy any living chess player, and using chess engines to analyze game play is a requirement to play at high levels.

  • Yes please. (Score:4, Informative)

    by Petersko ( 564140 ) on Tuesday December 12, 2023 @12:23PM (#64076193)

    If I got to choose, I might opt for the AI over the radiologist. Not that there aren't good radiologist... but if I've learned anything in the first five decades of my life, it's that in every single field, across the board, there's the same bell curve of competency. Doctors, lawyers, mechanics, coffee roasters... radiologists... there's a bunch of average people jammed up in the middle, some exceptional folks, and some people that have somehow managed to keep their jobs despite being woefully inadequate.

    At this stage in the technology, x-ray analysis seems like a pretty safe bet to at least get "solid" performance, if not stellar. I'll take it.

    • Hello, I work in radiology. I strongly recommend that you not opt for the AI over the radiologist at this point and time. Especially given the scant detail of this study, whose summaries includes nothing about the population studied, which conditions, etc. We are still struggling to get AI to identify COVID using large databases created with federal dollars in the US. Don't get your hopes up too fast.

      • These are very different use cases. One is a nicely contained problem with an absolute ocean of positive and negative diagnoses and outcomes, and a lot of work under the belt. The other (COVID) is almost completely dissimilar.

        The other difference is that from everything we know at this point, the X-Ray diagnosis is working.

        • It's AI trying to detect COVID (known outcome) from X-ray (same modality). So it's not dissimilar. That margins are what matters, not whether there is something obvious like a broken rib or a pneumothorax.

          • What makes it dissimilar is the quantity and quality of the training data. Your corner case - COVID - is relatively new, while there exists mountains of great training samples of all outcomes for the field of x-ray diagnosis in general.

            • X-ray is fairly harmonized, the other imaging modalities not at all. Garbage in, garbage out. Additionally, COVID, while new, still has a lot of data out there to train on, and it just isn't cutting the mustard yet. We don't know what a lot of DL algos are running off of, and there doesn't seem to be much interest in figuring it out, only in getting results that we can generate AUCs for, and then say, "Oh look, this number is higher, so it's better, right?" Shades of Spinal Tap abound in Radiology.

  • The way humans learn how to interpret X-Rays is by looking at a bunch of images and forming some "rules" in their head.
    How do they find those rules, by corroborating with other tests. Like if you see a tumor, you then run a test and verify that it is malignant.
    Such a process in humans is slow, tedious, and error prone.

    Such a process with a statistical machine like a Convolutional Neural Network is going to be able to form much better rules than some informal learning that has slowly developed across doctors

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      Basically, but it's more interesting than that.

      Students are usually given a small set of rules to start with. Some of these are new and well tested, some less so. Then they look at images and get the yes or no from their instructor. Eventually, when they get good at it, they're faster, more accurate, and don't follow he diagnostic criteria they started with.

    • About ten years ago, my Primary, a highly experienced nurse practitioner, didn't like the looks of a new blemish on the side of my forehead. Being careful, she set me up with a Dermatology consult. The consult was with a Resident, who had a book open to a page filled with photographs of the type of skin cancer she thought it might be. After comparing the blemish with the photos, and deciding that it didn't look malignant, he called in his attending who did her own comparison and agreed with him. What th

  • In a few years, AI will get much, much better. Eventually doctors will be freed to do other stuff, since their time is at an extreme premium, especially since COVID killed off 20% of them. ;-(

    • Re: (Score:2)

      by evanh ( 627108 )

      It works until it doesn't. Then it spits out garbage. If there isn't someone with understanding using the machine then the death rate will skyrocket.

      • Doctors have that problem, too. They miss stuff, or they make false positives. The problem is if doctors miss something you can't sue them.

        • Re: (Score:2)

          by evanh ( 627108 )

          Not even close. A doctor will say when something is unknown to them. That happens lots. A so-called AI has not the slightest clue, it will just pick something, no exceptions. Watson was a massive failure in this very application.

  • This is what really scares me about the rush to put these algorithms into medical care:

    Companies will progressively cut humans out.
    Companies will progressively turn over the decisions to software
    The people, through their government, will, finally put in some anti-WOPR law that it still has to be finally approved by a human.
    Companies will task some poor schmuck to sit at a computer and sign-off on the algorithm's decision -- oh but that person's job metrics will be based on the expectation that they can proc

  • "Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective, multicentre study" Lancet Digital Health https://doi.org/10.1016/S2589-... [doi.org]

  • My father is a radiologist. When he was taking his boards, the examiner put up an x-ray and my dad said, "This is a chest x-ray of a female patient." The examiner stopped him and said, "How do you know the patient is female?" because the patient had a double mastectomy so there was no breast shadow. My dad said, "I can tell by the width of her watch band." Would an AI know to think outside the scope of its training? Maybe. It would be an interesting experiment.

    On a related note, I once turned on the TV sho

    • Re: (Score:2)

      by dvice ( 6309704 )

      > Would an AI know to think outside the scope of its training?

      No, but there is no reason why you can't include watch information to the training set. We already have a case where "An artificial intelligence system is able to identify the biological sex of people just by reading their eyes."

      Random source for the story:
      https://www.infobae.com/en/202... [infobae.com]

    • If your father is making medical determinations based on somebody's accessories, your father has no business practicing medicine.

  • "more than 1,400 X-rays analysed by the software were cross-examined by senior radiologists."

    I would have liked to have seen that grilling. Did the x-rays wilt under cross-examination? Did they break into tears? Plead the 5th? Oh wait, no 5th in UK.

  • About 15 years ago I woke up with have my face paralyzed. I saw a neurologist, who looked at me for 30 seconds, told me I had Bell's palsy, and more or less waved me away. I had Ramsey Hunt Syndrome. And so I completely missed the window in which some treatment could have helped.

    Now, anecdote is not data. Except that yes, the data shows that missed diagnoses are common. And doctors are just educated people - and they fuck up all the time, living in the perfectly normal paradigm we all inhabit - the implicit

  • AI is a godsent for medical use and I cannot wait until the tech becomes sufficiently good that it will provide us TRUE proactive health care.

  • people, even doctors, are inattentive numpties at best

  • Assume the current AI tool is as good or better than human. What happen in the future when an update of the algorithm or tool, an improvement or twist or fix, leaves a corner case terribly wrong for a small group of patients? AI is the ultimate group-think.

  • Last spring, I had an MRI. I get them every six months because I had liver cancer seven years ago. It, the AI program, found a pea-sized tumor and rated it a 5, definitely cancer. The doctors had a conference and decided that it must be cancer, even though none of them would have classified it as such themselves. They ablated it, fried it with microwaves, and I'm cancer-free again. I hope. An AI bot can scan quicker than a doctor can and is a very useful tool.

Slashdot Top Deals

God help those who do not help themselves. -- Wilson Mizner

Close