Stanford Trains AI To Diagnose Pneumonia Better Than a Radiologist In Just Two Months (qz.com) 75
A new paper from Stanford University reveals how artificial intelligence algorithms can be quickly trained to diagnose pneumonia better than a radiologist. "Using 100,000 x-ray images released by the National Institutes of Health on Sept. 27, the research published Nov. 14 (without peer review) on the website ArXiv claims its AI can detect pneumonia from x-rays with similar accuracy to four trained radiologists," reports Quartz. From the report: That's not all -- the AI was trained to analyze x-rays for 14 diseases NIH included in the dataset, including fibrosis, hernias, and cell masses. The AI's results for each of the 14 diseases had fewer false positives and false negatives than the benchmark research from the NIH team that was released with the data. The paper includes Google Brain founder Andrew Ng as a co-author, who also served as chief scientist at Baidu and recently founded Deeplearning.ai. He's often been publicly bullish on AI's use in healthcare. These algorithms will undoubtedly get better -- accuracy on the ImageNet challenge rose from 75% to 95% in just five years -- but this research shows the speed at which these systems are built is increasing as well.
Volume might be the issue (Score:5, Insightful)
From what I have heard, most radiologists review large stacks of MRIs, CTs, or Xrays. They miss stuff all the time. AI wouldn't get tired, or be in a hurry.
...in training too. (Score:5, Insightful)
Volume also plays a role in training too.
To be considered trained, the radiologist usually have to go through several dozen of hundreds of MRIs, CTs and Xrays.
(They are not litteraly counted one by one. It's just accepted estimation that by the time the medical doctor finishes 5 years intership, he'll have seen enough example to be considered trained enough to have his radiologist certification).
The big advantage of the machine, is that instead of taking 5 years of internship training, you can have the neural net train by going through the 100'000 in one big computational jobs on the cluster.
There's this folk saying (Started by Malcolm Gladwell) that you need 10'000 hours of practice to become a master of anything.
The big benefits of AI is that these 10'000 hours don't need to happen in real-time anymore but can be simulated in a computer.
Re: (Score:3)
Also, once you've trained one machine, it's easy to transfer that knowledge to another computer. It's also easy to correct mistakes. Let's say that a computer had a mistake. It's easy to feed that information back into the system and make it learn from it's mistakes. with radiologists, even if you did let them know about the mistake, it's not so certain that they would learn from that mistake, and they might even take it the wrong way, because people have egos and feelings.
Re: (Score:3)
Not discounting the usefulness as a tool, just adding perspective to the comparison.
Re: (Score:2)
Did they take into account the trained conservatism in diagnosis? Radiologists (and doctors in general) will err on diagnosing as possible positive and then testing for validation. AI doesn't care about playing it safe.
I think you'd still want human judgement in the loop, at least in the near term. However, there's no reason the AI couldn't be trained to emit "probably X, do tests A, B and C to confirm".
Re: (Score:2)
better yet, a result and a confidence figure. Most likely, high confidence results just accepted and a human review for more marginal cases.
Re: (Score:2)
And speaking of mistakes - possessive of "it" is "its" (no apostrophe). "It's" is a contraction of "it is".
Re: (Score:2)
THANK GOD you caught that terribly confusing mistake before it killed billions!!! That was, of course, the most important aspect of the discussion. People dying (or not) of cancer pales in comparison.
Re: (Score:2)
It's also easy to correct mistakes. Let's say that a computer had a mistake. It's easy to feed that information back into the system and make it learn from it's mistakes. with radiologists, even if you did let them know about the mistake, it's not so certain that they would learn from that mistake,
It's that simple.
1. Feeding back a mistake may or may not change the model. It probably won't. When you train a machine learning model, it finds the best fit which still fails to match some of the input (i.e., you don't typically get 100% accuracy even on the training set)
2. A sufficient amount of new data that does change the model could break cases that were previously accurate. Humans don't typically re-consider past correct decisions just because they learned something new. A machine learning algorith
Re: (Score:1)
Medicine is too broad a subject to leave to humans (Score:5, Insightful)
It's becoming increasingly clear that no human can possibly have a functional grasp of all the knowledge required to make accurate diagnosis across all possible conditions. In the current model patients hope that they have something simple or obvious, and if not that their doctor can send them to the correct expert. This has far too many false positives and false negatives built in.
AI systems able to access comprehensive libraries of information are better at this type of work. Sure, I'd want an expert who can tailor search terms, accurately describe symptoms in a consistent manner, but for a number of years now I've been cheering every AI advance in clinical diagnosis. Can't come soon enough.
Much more complex (Score:4, Interesting)
It's becoming increasingly clear that no human can possibly have a functional grasp of all the knowledge required to make accurate diagnosis across all possible conditions.
No, it's much more subtle than that.
The things that they teach you at med school, are mostly rule of thumb. Simple algorithms that you can learn so you can get through your job without killing too many patients.
To diagnose pneumonia, there's a check list of things that you learn to look for and which allow you to say with some relative certainty whether or not the thing you're seeing is pneumonia.
Then there's the "clinical" experience. After seeing things for thousands of times, you start recognizing them automatically.
Like looking at an X-Ray picture and immediately "feeling" that there's "something funny" without even needing to start going through any checklist.
It's your old family doctor who can automatically guess the problem just be looking at how you walk like entering his office, or just based on the noises he hears outside, from the waiting room.
(Of course, there's some part of Sherlockian lightning fast thinking and deducting going on and a strong focus on very small otherwise imperceptible details.
But there's also some part of instinctive almost-sub-conscious gut feeling - how else would you know on *which* of the thousands of small imperceptible details to focus your inner Sherlock on ?)
That not something that you can learn by rote memorization during medical studies. That's something that comes slowly over time with practice.
The big advantage of neural nets, is that you can simulate all this experience inside a computer, by "simply" throwing hundreds of thousands of pictures at the neural into in a huge computational batches on the cluster.
AI systems able to access comprehensive libraries of information are better at this type of work. Sure, I'd want an expert who can tailor search terms, accurately describe symptoms in a consistent manner, but for a number of years now I've been cheering every AI advance in clinical diagnosis. Can't come soon enough.
In the end, as any other advances in the artificial intelligence field, you'll still need human oversight in the foreseeable future.
AI currently isn't replacing the job of actual doctors, as it is in providing more information faster to help taking the decision while taking all other informations on the way.
(Just like ECG able to propose diagnostic didn't cause the cardiologist specialist to disappear over-night).
AI Irony (Score:2)
The things that they teach you at med school, are mostly rule of thumb. Simple algorithms that you can learn so you can get through your job without killing too many patients.
Which, by the way, is going to be an irony completely lost on the "replace occurence of 'AI' with 'algorithm' and complain loudly" trolls that invariably pop-up on each machine learning article.
In, this case, it's the fresh med-school graduate who's using "an algorithm", and the AI which is most definitely relying on the intrinsic pattern-finding properties of any neural network / brain (be it in a biological real-world brain or a simulated one).
Re: (Score:1)
I'll add to this that, frankly, modern doctors by and large suck, or maybe they're overwhelmed by insurance forms or the medical examination chain is broken because you no longer spend 20 minutes with your doctor and instead spend 6 with a receptionist and then 7 with a nurse and then another 6 with a nurse practitioner, and maybe 1 minute with an actual doctor.
A couple of years ago I was reading an article about common diagnoses that doctors miss, and at the end of the article was an exam where they gave a
Re: (Score:1)
Actually it's much more complex than that.
Unless it's lung-drowningly obvious, pneumonia is a clinical diagnosis, not a purely radiographic one.
Perhaps it's some dork's wet dream to have all-knowing AI make all their decisions in life for them, but I can't help wondering how much humanity we're willing to give up.
Rules (Score:2)
Even more complex you may not know what rules the AI is using to evaluate something.
Just as the old doctor with 30 years of experience, won't actually know how/why/which neuron are firing up when he gets his hunch.
But he can then Sherlock it to gets a reasonable justification why he should trust his hunch.
The same here : the AI give its best idea about the X-ray.
Then the doctor who's seeing the patient will combine this information with what the rest (patient's complaints, etc.) and make a diagnostic and take a therapeutic decision.
Maybe the AI's diagnostic makes 0% clinical sense, in whic
Re: (Score:3)
It's becoming increasingly clear that no human can possibly have a functional grasp of all the knowledge required to make accurate diagnosis across all possible conditions. In the current model patients hope that they have something simple or obvious, and if not that their doctor can send them to the correct expert. This has far too many false positives and false negatives built in.
AI systems able to access comprehensive libraries of information are better at this type of work. Sure, I'd want an expert who can tailor search terms, accurately describe symptoms in a consistent manner, but for a number of years now I've been cheering every AI advance in clinical diagnosis. Can't come soon enough.
The AI can also triage the scans into sets based on diagnosis which would allow the radiologist to verify the most serious cases or positive results first, and then validate the negative results. This would be better use of their time while still providing a check on the AI. Better still, negative results could be reviewed by a trained NP or PA and the questionable ones sent to an MD; saving money in the process without reducing the quality of care.
Re: (Score:2)
False positive and false negatives are both dangerous and hence need to be reviewed. I do not see how the AI can be used to reduce cost. It can increase accuracy and speed of positive detection when used in parallel with trained processionals. It's like having a second opinion.
Re: (Score:2)
False positive and false negatives are both dangerous and hence need to be reviewed. I do not see how the AI can be used to reduce cost. It can increase accuracy and speed of positive detection when used in parallel with trained processionals. It's like having a second opinion.
It's a matter of who review what; having an NP or PA review all the negative results first and only forwarding the questionable ones to an MD would be far cheaper than having an MD review all of them. Part of the problem, however, is the notion you must always use an MD rather than an NP or PA for the entry point into care or to handle cases. NP already do various types of care that MDs do, such as anesthesiology and psychiatry, and sometimes independently; it's a matter of training them to perform specific
Re: Medicine is too broad a subject to leave to hu (Score:2)
Re: (Score:2)
It's becoming increasingly clear that no human can possibly have a functional grasp of all the knowledge required to make accurate diagnosis across all possible conditions. In the current model patients hope that they have something simple or obvious, and if not that their doctor can send them to the correct expert. This has far too many false positives and false negatives built in.
AI systems able to access comprehensive libraries of information are better at this type of work. Sure, I'd want an expert who can tailor search terms, accurately describe symptoms in a consistent manner, but for a number of years now I've been cheering every AI advance in clinical diagnosis. Can't come soon enough.
The thing is, it's not necessary for a medical AI to know everything or have a 99.99999% success rate, unlike an autonomous car which has to make split second decisions based on incomplete information.
A medical AI analyses a sample and says "Hey doc, this looks like Cancer", the doctor then has a look and determines if it looks like cancer and determines what extra tests are needed or if it's clear enough, goes straight to recommending a treatment. At the very least, it provides an extra set of eyes.
I
Old tech and idea (Score:3, Insightful)
They've had this sort of tech to read digital images for decades. Papnet is used to read pap smears since the mid-1990s.
http://www.lightparty.com/Heal... [lightparty.com]
Re: (Score:3)
They've had this sort of tech to read digital images for decades. Papnet is used to read pap smears since the mid-1990s.
http://www.lightparty.com/Heal... [lightparty.com]
That's like saying the Webb Space Telescope is old tech because Sputnik. Papnet can't read x-rays, and it's not because it never occurred to anyone to try to automate the analysis 20 years ago, it's because it's a harder problem that 90s-era technology couldn't handle.
Yes let the AI decide. (Score:5, Interesting)
I had unnecessary lung surgery 12 years ago because I was misdiagnosed with lung-cancer when all I had a mild fungal infection common in my area at the time called histoplasmosis. I’ve no doubt AI’s would prevent these kinds of mistakes. I also remember being told my chance of having lung cancer was 97% by the radiologist. I’m pretty sure as a young non-smoker he didn’t factor in any kind of bayesian statistics to arrive at this answer, but just looked a chart based on signal return from a PET scan. Hopefully these days at a minimum there would be some app that has them plug in relevant variables and does the bayesian analysis for them.
Trust me, recovery from invasive lung surgery is no picnic.
We don't get bent out of shape when machines read bar codes to price our grocery items instead of a human reading the price sticker. The machine is doing essentially the same thing, but far more accurately. The only difference is we take the price and refactor into a visual form easier for the machine to read. Imagine the accuracy if the Human had to read the bar code instead and remember the item associated with it.
Perfect the Enemy of the Good (Score:2)
Sooooo...... apparently better that 9 people undergo unnecessary surgery, because the computer would make a mistake on the 10th also. If it's not perfect, then we'd better rely on the humans.
Remember, do not let the Perfect be the enemy of the Good
Same comment would apply to Self Driving Cars, which I'd bet you are also against.
Re: (Score:2)
From TFA, the computer is already better than the experts. It will only improve from this point forward. Does it matter if it 10% better or 10 times better. Lives will be saved. The 9 out of 10 was merely illustrative. If only 10% better now it will likely be 10 or even 100 times better ten years from now. We are at what is known as a tipping point. You're attitude seems to suggest computers will never be as good as humans at this activity, can't be trusted, and you would prefer humans to do this kind
Critical out of band information (Score:1)
Is the AI allowed to know what medical insurance the patient has, in order to extend the recommended treatment just enough to keep all the hospital beds occupied? I swear, I've run into this. When I turned up with chest pain, and good insurance, they took three days of running me through all the expensive tests, including the machine that goes "ping!"
My own non-specialist explained it in 30 seconds with a good listen to my chest when I got to him. Pericarditis: there's a very characteristic rubbing sound,
Re: (Score:2)
Let's say they didn't: (and this same type of back and forth q and a I have observed of attendings teaching resident's with the below line of questioning
1)patient discharged wi
It's Deep Learning (Score:3)
Re: (Score:2)
"We develop an algorithm..."
What does "develop" and "algorithm" mean in context of deep learning?
Is it an "algorithm" or a process of optimizing the depth of the net?
And by "develop" do they mean they selected the best performing NNs structure as I suppose the process of "training" it is automatic?
Re: (Score:2)
The resulting network is the algorithm. "Develop" usually means "propose a specific neural network architecture" in this context. So no, no meta-learning, nor novel optimization algorithm.
Re: (Score:2)
The resulting network is the algorithm. "Develop" usually means "propose a specific neural network architecture" in this context. So no, no meta-learning, nor novel optimization algorithm.
OTOH, choosing a good neural network architecture makes a huge difference, and is decidedly non-trivial.
If it takes.... (Score:2)
Re: (Score:2)
Doctors are history
In the long run we are all dead. But in the next few years *radiologists* are going to find themselves in serious trouble I think.
Radiology is a highly paid specialty, so the incentive to automate it is there.
Much of the work consists of sitting in a dark room reading images that were acquired somewhere else. No need for manual dexterity or going from room to room, mostly no need for bedside manner. The closest most patients come to interacting with a radiologist is having the primary care doctor show th
Testing (Score:2)
You have to be careful that you haven't inadvertently over-trained the AI so that it's basically memorizing the individual data points rather than reaching generalizations. You can leave some out to test against, but this can inevitably be trained against in an evolutionary sort of way.
The only way to know for sure is to take the "finished product" and run it against brand new stuff it has never seen before, once.
false negatives (Score:2)
False negatives are dangerous if this was to be used to minimize human's effort (ie first line of analysis). ...and this is likely of how it was trained as they used known database of Xrays and correlate it with known positives.
Can they tell the probability of false negatives (ie missing a problem) given the training set?
So likely it could only be used in addition to human's analysis.
I have an epistemological problem here. (Score:2)
How would we *know* whether the AI is better than humans? How do you *know* the AI's false positive and false negative rate without some kind of oracle to compare the result to? Can an AI algorithm somehow outperform the human-made classifications it is trained on, and if it appears to do so should we trust that result?
I'm not doing the usual Slashdot thing of assuming that experts are too dumb to see the objections that immediately occurred to me; I just think that people take a headline like above for g
Re: (Score:2)
Have you tried reading the article ?
Re: (Score:2)
Yes.