Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Medicine The Internet Communications Earth Google Network Science

Doctors Perform Better Than Internet Or App-Based Symptoms Checkers, Says Study (sciencedaily.com) 192

Posted by BeauHD from the captain-obvious dept.
An anonymous reader quotes a report from Science Daily: Increasingly powerful computers using ever-more sophisticated programs are challenging human supremacy in areas as diverse as playing chess and making emotionally compelling music. But can digital diagnosticians match, or even outperform, human physicians? The answer, according to a new study led by researchers at Harvard Medical School, is "not quite." The findings, published Oct. 10 in JAMA Internal Medicine, show that physicians' performance is vastly superior and that doctors make a correct diagnosis more than twice as often as 23 commonly used symptom-checker apps. The analysis is believed to provide the first direct comparison between human-made and computer-based diagnoses. Diagnostic errors stem from failure to recognize a disease or to do so in a timely manner. Physicians make such errors roughly 10 to 15 percent of the time, researchers say. In the study, 234 internal medicine physicians were asked to evaluate 45 clinical cases, involving both common and uncommon conditions with varying degrees of severity. For each scenario, physicians had to identify the most likely diagnosis along with two additional possible diagnoses. Each clinical vignette was solved by at least 20 physicians. The physicians outperformed the symptom-checker apps, listing the correct diagnosis first 72 percent of the time, compared with 34 percent of the time for the digital platforms. Eighty-four percent of clinicians listed the correct diagnosis in the top three possibilities, compared with 51 percent for the digital symptom-checkers. The difference between physician and computer performance was most dramatic in more severe and less common conditions. It was smaller for less acute and more common illnesses.
This discussion has been archived. No new comments can be posted.

Doctors Perform Better Than Internet Or App-Based Symptoms Checkers, Says Study More

Doctors Perform Better Than Internet Or App-Based Symptoms Checkers, Says Study

Comments Filter:

  • This doesn't prove what they were hoping to prove (Score:5, Interesting)

    by Applehu Akbar ( 2968043 ) on Wednesday October 12, 2016 @10:40PM (#53067129)

    "Eighty-four percent of clinicians listed the correct diagnosis in the top three possibilities, compared with 51 percent for the digital symptom-checkers. The difference between physician and computer performance was most dramatic in more severe and less common conditions. It was smaller for less acute and more common illnesses."

    I'm surprised that digital diagnosis is that good already. The era of an "iDoc" app being as good as a gateway practitioner is probably not far off.

    • Re: (Score:2)

      by Jzanu ( 668651 )
      You might want to reread all of that "more than twice as often" means 1/3~33% accuracy for the entire group of 23 symptom checking computer programs vs 2/3~66% accuracy for the doctors. Machines have the advantages of pure data processing, so this result shows that instant recall and effectively infinite knowledge bases still don't measure up to the cognitive processes performed by trained medical doctors during diagnosis.

      • Re: (Score:1)

        by Jzanu ( 668651 )
        And to be pedantic "more than..." suggests that the group of programs did worse than 1/3 correct diagnoses.

    • Re:This doesn't prove what they were hoping to pro (Score:5, Interesting)

      by demonlapin ( 527802 ) on Thursday October 13, 2016 @12:00AM (#53067427) Homepage Journal
      I'm a doctor, though not a diagnostician. Diagnosis is rarely hard - there are some hard cases, but they really mostly aren't. Do you have a persistently elevated blood glucose level? You have diabetes. Do you have consistently high blood pressure? You have hypertension. Etc. It's hardly surprising that computers are just as good as humans at diagnosing diseases that are mostly defined by strict, objective criteria.

      What is harder is management - finding the right collection of drugs that will effectively treat a patient's diseases without introducing too many side effects. And what's even harder is anything procedural - we have no computers that can actually do procedures at all. Those aren't what most people think of as "going to the doctor", but it's what most doctors do - either manage disease, or do procedures, both of which are either mostly or severely beyond the ken of computers. Show me a computer that can do something as simple as put in an IV, and I'll be greatly impressed. So many subtleties boil down to "well, I saw something once that looked just like this, and the solution was X..." that it's worth trying X before going on to Y and Z.

      My wife is a diagnostician - a neurologist. She sees stuff on a daily basis that would flummox any non-neurologist (really, I barely know what she's talking about half the time, and my peers would be much, much worse at that), let alone a computer. As the old joke goes, it's like being a car mechanic - who has to work on the car while it's doing 70 miles per hour down the highway, with zero downtime acceptable.

      • I'm a doctor, though not a diagnostician. Diagnosis is rarely hard - there are some hard cases, but they really mostly aren't. Do you have a persistently elevated blood glucose level? You have diabetes. Do you have consistently high blood pressure? You have hypertension

        You do realise that diabetes and hypertension are not real diagnoses?
        Diabetes does not describe the types of diabetes a person. Type I, Type II, other rarer types. All diabetes means is that you have elevated blood glucose levels.

        Ditto for hypertension.

      • Re:This doesn't prove what they were hoping to pro (Score:4, Insightful)

        by serviscope_minor ( 664417 ) on Thursday October 13, 2016 @04:37AM (#53068137) Journal

        Do you have consistently high blood pressure? You have hypertension.

        That's not really a diagnosis. That's just a different name for the symptoms. Bonus points for diagnosing "Pirmary Hypertension" which of course means "yeah dunno".

        • A very specific diagnosis is, after all, just another name for a list of the symptoms; you're just complaining that the list isn't precise enough. How much are you willing to spend to try to figure out precisely what's causing it? It's not so much "yeah, dunno" as "yeah, not worth trying to figure it out".

          I mean, there are lots of genetic mutations with "variable penetrance". Why do some people get just a touch, and others get slapped down hard? Could be auxiliary genes, could be genetic mosaicism, could

          • A very specific diagnosis is, after all, just another name for a list of the symptoms; you're just complaining that the list isn't precise enough.

            Not really. I was complaining about your *specific* choice. High blood pressure and hypertension mean the same. All you've done is given the symptom a different name.

            Compare to, you go in with a persistent, nasty sore throat and the doctor tells you have strep throat. There are many underlying causes of sore throats, such rhinovirii [*], smoke, excessive voice use

      • I'm a doctor, though not a diagnostician. Diagnosis is rarely hard - there are some hard cases, but they really mostly aren't. Do you have a persistently elevated blood glucose level? You have diabetes. Do you have consistently high blood pressure? You have hypertension. Etc. It's hardly surprising that computers are just as good as humans at diagnosing diseases that are mostly defined by strict, objective criteria.

        Excellant point. My lay viewpoint on what most doctors do is not diagnose, but treat symptoms, especially at the primary care level. Someone comes in displaying X, the thought process is X is treated most commonly by doing Y and hence we will try Y. If the symptoms disappear than the patients is cured and all is well. The doctor may also offer a diagnosis, often because the patient expects it, but finding the cause is less important than deciding what will make the symptoms go away. To extend your car an

      • Re: (Score:2)

        by clovis ( 4684 )

        The other problem is that some patients will lie about their symptoms -either exaggerating or denying. Doctors know this and have the problem of seeing through what the patient says without pissing the patient off so that the patient would then refuse to cooperate or comply.
        It will be interesting to see how an AI will deal with that aspect of medical care. And yes, I'm aware that there are tests, but you can't run every test just because.

        • I usually have the opposite problem. I can't identify or understand my symptoms, and often tell the doctors I need to find counseling and dig around a bit; they try to diagnose me anyway. That's why I have a diagnosis for anhedonia as a symptom of ADHD (as opposed to depression), which is correct; but I also have no idea how to measure treatment, which is bad. The drugs I'm on (amphetamine) have made me more-responsive, more outgoing, more engaged, and generally identifiable as a happier and more-emotio

      • Re: (Score:2)

        by Lennie ( 16154 )

        Your IV example is not a good example, that is actually something people are working on, here some quick list of some links:

        https://www.sciencedaily.com/r... [sciencedaily.com]

        https://www.youtube.com/watch?... [youtube.com]

        https://web.stanford.edu/group... [stanford.edu]

        http://www.yissum.co.il/techno... [yissum.co.il]

      • What is harder is management - finding the right collection of drugs that will effectively treat a patient's diseases without introducing too many side effects. And what's even harder is anything procedural - we have no computers that can actually do procedures at all. Those aren't what most people think of as "going to the doctor", but it's what most doctors do - either manage disease, or do procedures, both of which are either mostly or severely beyond the ken of computers. Show me a computer that can do something as simple as put in an IV, and I'll be greatly impressed. So many subtleties boil down to "well, I saw something once that looked just like this, and the solution was X..." that it's worth trying X before going on to Y and Z.

        The short answer is that a nurse with an expert system can do a better job of all that than a doctor without one... if the system is trained by competent doctors and statisticians.

        • I understand the POV you're espousing here - but the reality is that although nurses and doctors work in what seems like the same field to the layman, the reality is that the training regimens and skill sets are completely different. I've lost track of how many times I've had to explain things to very good, very experienced nurses that I would absolutely crucify a third-year medical student for not knowing.

          In theory, you're totally right. In practice, the nurses don't know what questions to ask, or how to

          • If you had a computer that was as good at interrogation as a doctor, we wouldn't have TSA agents.

            The TSA has never caught a terrorist and middle-aged crazy women manage to get through security and get on planes. The TSA is a jobs and harassment program, it is not about security.

            • Way to miss the point.

              • Way to miss the point.

                Uh no. See, you spewed some crazy in the middle of your comment there, and I corrected it. If you want me to believe you're working on logic, you're going to have to pick an example that isn't literally insane.

    • Re: (Score:2)

      by lorinc ( 2470890 )

      If this follows the same trend as we saw in computer vision in the last few years, then doctors will be outperformed by machines in less than a decade in all the simpler tasks. The thing is, we truly are only in the beginning of the era of machine learning, and currently, there is no upper bound to what it can possibly do.

      • we truly are only in the beginning of the era of machine learning, and currently, there is no upper bound to what it can possibly do.

        Of course there is. Machine learning can't change the laws of physics, for example, so we can use that as a starting point, and then use linear regression and K-means clustering to.. oh god damnit.

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      These were internet diagnosis apps, designed essentially as novelties to get ad revenue.

      Both Google and IBM are designing diagnostic systems for real. It will be interesting to see how they do.

    • "Eighty-four percent of clinicians listed the correct diagnosis in the top three possibilities, compared with 51 percent for the digital symptom-checkers. The difference between physician and computer performance was most dramatic in more severe and less common conditions. It was smaller for less acute and more common illnesses."

      I'm surprised that digital diagnosis is that good already. The era of an "iDoc" app being as good as a gateway practitioner is probably not far off.

      I was surprised that the doctor difference was smaller for common illnesses. I would have guessed the opposite. I would have expected a properly written AI to be much better at identifying obscure illnesses than a doctor that has never seen that particular illness. Also, they are using off the shelf symptom checker apps. Some of them might be ok but many are likely crap. If they are already at 51% then if someone like google seriously tried to tackle it with a decent budget then likely they could do mu

    • It actually sounds like people to me.

      You know maybe you're retarded, or have aspergers. You should get some ritalin. I can tell because you disagree with me and a lot of aspies disagree with me. /inarguable Fark diagnosis

      Seriously, though, people who have nfc about medicine (or any field) will recognize a set of familiar things and tie it to their experience. Their experience is going to be the most common thing anyone encounters, so they'll identify what looks vaguely like that as that.

    • Re: (Score:2)

      by clovis ( 4684 )

      "Eighty-four percent of clinicians listed the correct diagnosis in the top three possibilities, compared with 51 percent for the digital symptom-checkers. The difference between physician and computer performance was most dramatic in more severe and less common conditions. It was smaller for less acute and more common illnesses."

      I'm surprised that digital diagnosis is that good already. The era of an "iDoc" app being as good as a gateway practitioner is probably not far off.

      It's not that surprising when you consider that most of the diagnosis was done by the humans that chose what questions to ask and what tests to run before the data was presented to the AI. When the AI correctly chooses what questions to ask and which tests to run, then we'll have something.

    • ""Eighty-four percent of clinicians listed the correct diagnosis in the top three possibilities, compared with 51 percent for the digital symptom-checkers."

      I'm surprised that digital diagnosis is that good already.

      When comparing percentages, you need to compare them in the manner which has the biggest consequences. A good example is OCR software (optical character recognition). If one has an accuracy of 99.99% and another cheaper one has an accuracy of 99.95%, you might think there's very little diffe

  • flawed study (Score:5, Insightful)

    by Anonymous Coward on Wednesday October 12, 2016 @10:46PM (#53067161)

    In the study the doctors knew they had to perform well. In the real world you're lucky if they even listen to you for two minutes before prescribing what ever the pharma rep recommended at the free lunch yesterday

    • In the study the doctors knew they had to perform well. In the real world you're lucky if they even listen to you for two minutes before prescribing what ever the pharma rep recommended at the free lunch yesterday

      If that's the service you get, find a better doctor. There are good ones out there.

      The main thing to look for is a physician who takes the time to listen to you and to explain his reasoning and approach, and is willing to discuss the pros and cons of various alternatives with you. Ask a lot of good questions and you should get a serious, thoughtful and thorough answers that don't attempt to shut you up by burying you in technical language. If that's not your experience, find another doc, and keep looking

    • I install and repair computer systems for doctors at private practices. Every single one of them hates those pharma reps. I think you've concocted this fantasy where the reps bribe doctors with free lunches and golf trips. The reality is that they're door-to-door salesmen. Several of them come in every day, asking to talk to the doctor so they can give their sales pitch.

      The doctors hate this just like you hate door-to-door salesmen at your home, probably more so since they have to put up with a lot m

  • Observations (Score:5, Insightful)

    by darkain ( 749283 ) on Wednesday October 12, 2016 @10:48PM (#53067179) Homepage

    There is a hell of a lot more to observe with a patient than simple a checklist of yes/no values to see if someone has a particular diagnosis. For example, years back when I had a severe sore throat, I went into the doc. She took one look at me, mentioned there is a unique smell associated with strep throat, did the test for it, and handed me a prescription for the antibiotics all within a few short minutes. WebMD, as we all know, diagnoses cancer for when you stub your toe!

    • In this case, the human doctors only had a written description (I've posted examples in another comment.)

      However, the human doctor can use all the information in the description, whereas the app can only do so if it has a box for entering that information. E.g. "Preliminary laboratory studies are notable for a serum ALT of 6498 units/L, total bilirubin of 5.6 mg/dL, and INR of 6.8." (Actual text from one of the cases.) I expect apps aimed at consumers don't have any way to enter this information. (No, I don

      • To me only a written description is flawed. Both the computer and the doctor should be able to ask questions and request tests.

        That is where the computer would really shine. Digging into more obscure causes and spending more "time" with patients.

    • Comment removed based on user account deletion

      • Told me to drink less and that was it.

        This is one of the best treatments for a diagnosis of gout. That and drinking enough water.

      • So you told your doctor what was wrong, he looked at it, determined it makes sense, and went with it? Welp. Makes sense.

        I like that he handled the semantic pollution by finding someone who had nfc what might be going on and asking them to diagnose in a vacuum. Interesting technique.

    • Re: (Score:2)

      by ceoyoyo ( 59147 )

      Cases where there's a definitive test like that are a slam dunk for a computer system. See these symptoms, order this test, act on the result (yes, you, probably through a proxy, likely paid several hundred to several thousand dollars for that). The tough ones are where the signs and symptoms are vague or there isn't a simple test.

    • Re: (Score:2)

      by rhazz ( 2853871 )
      That's interesting, I had a rather opposite opposite experience with a doctor. Last year my infant son was diagnosed with strep (after throat swab) and went on antibiotics. Three days later, my wife and I went to the doctor because we also had strep-like symptoms, and the doctor had us take throat swabs also. My wife tested positive, I tested negative. So she got antibiotics and I got nothing. 2 weeks later my wife and son are healthy, and I am still suffering from worsening strep-like symptoms. So I went b

      • Overuse of antibiotics is a problem; but I would have assessed the risk of antibiotics versus the risk of a bad lab test or later contraction (exposure is obvious here) and leaned toward antibiotics as a preventative at least. I'm not a doctor, though; and there's probably a whole hell of concerns about preventative antibiotics. It messes with your gut flora, it has all the risks of normal use (supergerms!), etc. Plenty of reasons to wait on that; although also plenty of reason to consider the risk of a

  • The answer, according to a new study led by researchers at Harvard Medical School, is "not quite."

    Oh, well, that's okay then. Everyone, pack up your computers and smartphones; they're completely useless. Let the medical school researchers diagnose your conditions in the future, because this is the best it's ever going to be.

  • More LUDDITE lies! (Score:2, Funny)

    by Anonymous Coward
    ONLY apps can app apps, NOT LUDDITE doctors!

    Apps!

  • It's only a matter of time before that's not the case for the vast majority of diseases. Combine data (both historical and from diagnostics) and machine intelligence, then let someone collect a vast dataset of symptoms, diagnoses, treatments and outcomes and train some algorithms on it, and voila, you $300K med school degree is now mostly worthless. That'll make healthcare quite afforable, though, right up until you need something done that machines can't do yet. I don't believe human intervention is actual

    • When doctors are no longer necessary a lot of other people will no longer be necessary. I believe at that point we can all relax more and have 10 hour work weeks.

      • Actually my Universal Social Security proposal has a slight side-effect in that regard: it relieves a complex system of inefficiencies in the U.S. economy, and ends up raising consumer buying power beyond labor supply. That is to say: we end up with negative unemployment (-18% to -23% by my simple models).

        The immediate remediation is to make everyone poorer by reducing productivity by 20%. To do this, you'd reduce working hours to 32/week (4 days). That means each human being is only applying 32 hou

  • Example cases (Score:3)

    by Michael Woodhams ( 112247 ) on Wednesday October 12, 2016 @11:36PM (#53067355) Journal

    I managed to track down the actual text of the cases. TFA was only adding the human doctors to an analysis already done with the aps. The aps paper is http://www.bmj.com/content/351... [bmj.com] and the cases are in the supplementary material ('data supplement') http://www.bmj.com/highwire/fi... [bmj.com]

    A 48-year-old woman with a history of migraine headaches presents to the emergency room with altered mental
    status over the last several hours. She was found by her husband, earlier in the day, to be acutely disoriented and
    increasingly somnolent. On physical examination, she has scleral icterus, mild right upper quadrant tenderness, and
    asterixis. Preliminary laboratory studies are notable for a serum ALT of 6498 units/L, total bilirubin of 5.6 mg/dL, and
    INR of 6.8. Her husband reports that she has consistently been taking pain medications and started taking additional
    500 mg acetaminophen pills several days ago for lower back pain. Further history reveals a medication list with
    multiple acetaminophen-containing preparations.

    (This one is acute liver failure requiring emergency care).

    An 18-month-old toddler presents with 1 week of rhinorrhea, cough, and congestion. Her parents report she is
    irritable, sleeping restlessly, and not eating well. Overnight she developed a fever. She attends day care and both
    parents smoke. On examination signs are found consistent with a viral respiratory infection including rhinorrhea and
    congestion. The toddler appears irritable and apprehensive and has a fever. Otoscopy reveals a bulging,
    erythematous tympanic membrane and absent landmarks.

    (Acute otitis media - requires 'non-emergent care', i.e. needs professional medical care but is not an emergency)

    A 34-year-old woman with no known underlying lung disease 12-day history of cough. She initially had nasal
    congestion and a mild sore throat, but now her symptoms are all related to a productive cough without paroxysms.
    She denies any sick contacts. On physical examination she is not in respiratory distress and is afebrile with normal
    vital signs. No signs of URI are noted. Scattered wheezes are present diffusely on lung auscultation.

    (Acute bronchitis, self-care appropriate.)

  • They told my uncle he had six months to live, and he lasted for nearly seven. So much for the so-called "experts".

    And have you seen their handwriting? My five year old can do better.

    • Re: (Score:2)

      by Jzanu ( 668651 )
      If you judge medical expertise based on statistical outliers with undisclosed behavior and on handwriting quality, then you need to spend time in an asylum.

    • Re: (Score:1)

      by Anonymous Coward

      3 replies so far that fail to spot that your post was humorous.

      Here's a tip Slashdotters: If you're a stupid cunt, leave this site the fuck alone so I am not forced to wade through your moronic drivel that you mistakenly believe the world wants to hear.

  • How well do doctors compare to a patient with access to the internet and a good dose of motivation?

  • A Dr in a country practice will soon get to understand what most of their active patients present with.
    A Dr enjoying the normal, safe, happy suburbs will soon find their way around the over fed, middle-class life style aliments.
    Having to serve in the inner city slums will present the vast complexity of poverty, live style, work related exposure and drug related conditions combined with poor nutrition.
    Combine that with a flood of very sick people who bring with them rare, regionally eradicated and contagi

  • How did the humans do against the *best* of the algorithmic diagnosers?

  • I didn't catch the name of the apps they evaluated. In machine learning, the difference between cutting edge and older algorithms could be very large. If they compared AI to humans, they should at least compare to the best medical diagnosis AI.

  • Why is it the medical field gets paid for a incorrect diagnosis and the treatment as well as correct ones? I think performance would increase if they knew they wouldn't get paid or have to refund it. Right now they can walk in and just say "It's this" and walk out knowing they'll still get paid.

    • Why is it the medical field gets paid for a incorrect diagnosis and the treatment as well as correct ones? I think performance would increase if they knew they wouldn't get paid or have to refund it.

      Excellent point, but we can take it further. How about programmers only get paid when they produce bug-free code?

  • But my doctor has a much slower startup time than my health app.
    My app needs around 3 seconds while visiting my doctor needs an hour long drive and a 2 hour waiting room wait.

  • Self-diagnosis (Score:3)

    by Alomex ( 148003 ) on Thursday October 13, 2016 @07:19AM (#53068609) Homepage

    They are comparing doctor diagnosis vs. self diagnosis. It doesn't surprise me at all that doctors are better.

    However if we compare doctor vs. doctor&software the latter wins by a mile. The best diagnosis software out there is Isabel HealthCare [isabelhealthcare.com] with proven, peer-reviewed results.

  • Within our lifetime computers will replace the GP. That's where the lowest hanging fruit is. 1. Cheaper for insurance. This will drive the implementation, much as Uber pushes self driving cars. 2. 24/7/365 access in both good and bad neighborhoods. 3. GP bot only has to weed out to the base of the problem. If the case and treatment doesn't fall within the first line of medical defense, then GP bot just gives a referral to a human specialist. 4. One central aggregate database will lead to better care

  • An AI can't make mistakes, and only ignorant Luddites with severe paranoia issues would retard process and a better world without human error and the terrible death tolls that follow when doctors are texting or drunk or ... Oh wait. That's self-driving cars. Only SDCs are perfect, I guess, though one would think SD AI doctors would be far better than humans, given the premise of SDCs. If you trust an AI to drive a car, you should trust it to diagnose your cancer. Mistakes on either's part will kill you.

  • Mechanics are better at replacing car tires than mobile phones, notebooks and desktops. Allegedly being able to actually touch the subject matter proves a significant advantage.
  • Contrary to the claim in this "study", articles have been coming out over the past 6 years or so claiming just the opposite.
    I guess results depend on who is funding the study.Also, I'm sure there's zero bias from the AMA.either way on this topic
    http://caledonianmercury.com/2... [caledonianmercury.com]
    another from 2013: http://io9.gizmodo.com/5983991... [gizmodo.com]

    Yet another from 2013
    https://gigaom.com/2013/02/11/... [gigaom.com]

  • Just saw something on Watson and its achievements in the field of Medicine. Certainly an interesting tool that may change how medicine is practiced.

Slashdot Top Deals

Never test for an error condition you don't know how to handle. -- Steinbach

Close