Statisticians Warn That AI Is Still Not Ready To Diagnose COVID-19 (discovermagazine.com) 38
Discover magazine reports:
For years, many artificial intelligence enthusiasts and researchers have promised that machine learning will change modern medicine. Thousands of algorithms have been developed to diagnose conditions like cancer, heart disease and psychiatric disorders. Now, algorithms are being trained to detect COVID-19 by recognizing patterns in CT scans and X-ray images of the lungs.
Many of these models aim to predict which patients will have the most severe outcomes and who will need a ventilator. The excitement is palpable; if these models are accurate, they could offer doctors a huge leg up in testing and treating patients with the coronavirus. But the allure of AI-aided medicine for the treatment of real COVID-19 patients appears far off. A group of statisticians around the world are concerned about the quality of the vast majority of machine learning models and the harm they may cause if hospitals adopt them any time soon.
So far, their reviews of COVID-19 machine learning models aren't good: They suffer from a serious lack of data and necessary expertise from a wide array of research fields. But the issues facing new COVID-19 algorithms aren't new at all: AI models in medical research have been deeply flawed for years...
"The main problems appear to be (perhaps unsurprisingly) a lack of necessary data, and not enough domain expertise," writes Slashdot reader shirappu: The leader of the above group of statisticians, Maarten van Smeden, pointed to a lack of collaboration between researchers as a road block in the way of developing truly accurate models. "You need expertise not only of the modeler," he said, "but you need statisticians, epidemiologists and clinicians to work together to make something that is actually useful."
One biostatistician has even been arguing that with many current AI models, medical researchers are using machine learning to "torture their data until it spits out a confession."
Many of these models aim to predict which patients will have the most severe outcomes and who will need a ventilator. The excitement is palpable; if these models are accurate, they could offer doctors a huge leg up in testing and treating patients with the coronavirus. But the allure of AI-aided medicine for the treatment of real COVID-19 patients appears far off. A group of statisticians around the world are concerned about the quality of the vast majority of machine learning models and the harm they may cause if hospitals adopt them any time soon.
So far, their reviews of COVID-19 machine learning models aren't good: They suffer from a serious lack of data and necessary expertise from a wide array of research fields. But the issues facing new COVID-19 algorithms aren't new at all: AI models in medical research have been deeply flawed for years...
"The main problems appear to be (perhaps unsurprisingly) a lack of necessary data, and not enough domain expertise," writes Slashdot reader shirappu: The leader of the above group of statisticians, Maarten van Smeden, pointed to a lack of collaboration between researchers as a road block in the way of developing truly accurate models. "You need expertise not only of the modeler," he said, "but you need statisticians, epidemiologists and clinicians to work together to make something that is actually useful."
One biostatistician has even been arguing that with many current AI models, medical researchers are using machine learning to "torture their data until it spits out a confession."
Comment removed (Score:3)
Re: (Score:1)
Many of these models aim to predict which patients will have the most severe outcomes and who will need a ventilator.
Re: (Score:2)
Re: (Score:2)
I think the article is really saying we didn’t get the answers we wanted.
“Why isn’t everyone dying under this model? It’s all wrong! This AI doesn’t ship until Windows doesn’t run on OS2!”
Re: (Score:2)
Still not what the article is about. RTFA and get back to us.
Re: (Score:2)
Re: (Score:2)
It's not about who is at risk, in the general sense. Besides, the virus lacerates the insides of all age groups and that kills more than the lung infection.
It's about who is going to get which effect, which is totally different.
Re: (Score:2)
Besides, the virus lacerates the insides of all age groups and that kills more than the lung infection.
Would need an actual citation for that, because the CDC only has one rigorous tiny study [cdc.gov] of 8 patients that showed that. And it was for people from a nursing home, with a median age of 73.5 years old.
Re: (Score:2)
Funding.
The inevitable financial depression due to COVID-19 fast-forwards arrival to AI winter [wikipedia.org], so many AI shops that have been surfing on subsidies and promises of unicorns are now grasping at straws, as many clueless people giving them money out of fear of missing out are now realizing the results are just not there and won't be there either in a year or five.
Re: (Score:3)
So many companies have invested millions and millions into developing so-called 'AI' thinking it was Just Another Development Cycle, only to find out that it's more-or-less a dead end. Now they have investors and stockholders breathing down their necks, ready to chop their heads off if there isn't a Return on Investment. So their marketing departments are throwing the thing at every wall they can, seeing if it 'sticks' anywhere, and hypin
The point is this... (Score:2)
There's some young businessman wanting to make a name for himself who thinks, "AI is the answer to everything!"
And what none of these people get is that AI is only as good as the data that's fed it. A core component of any mathematical learning algorithm is its ability to compare its results against reference table data, then adjust its calculations to better match the results. But if your reference table data is junk, so too shall be your "AI". And there's the problem. To my knowledge, doctors have not
Yes, but you're in the wrong forest (Score:2)
This isn't about diagnosing if the person has COVID. Although with a 40% failure rate for most existing tests, that would be an improvement.
This is about determining how severe it'll get. There are seven different categories patients can fall in, knowing which in advance is a definite benefit.
Garbage in, garbage out (Score:1)
Dear domain experts (Score:1)
Re: (Score:1)
This is a great age we live in (Score:3)
AIs who sort of guess you might be sick - maybe - and POTUSes who "feel" a remedy will work.
I think we should just go back to leeches and perfurme: we'll have a better chance of getting cured.
Re: (Score:1)
Re: (Score:2)
Questions about the first article: (Score:1)
As usual, the last line of the summary is stupid (Score:2)
One biostatistician has even been arguing that with many current AI models, medical researchers are using machine learning to "torture their data until it spits out a confession."
You know what we call torturing data to create a model when we don't use machine learning? Science.
Machine learning (ML) models are simply a new kind of model. Yes, they are often opaque. Yes, they can occasionally give bad results. But they can also provide brilliant results that are otherwise inaccessible. Discounting these resu
Re: (Score:2)
You know what we call torturing data to create a model when we don't use machine learning? Science.
While that approach is so common today, it is, in fact - not science. Richard Feynman [youtube.com] lays it out quite nicely. You make a guess, you calculate the results of your guess, you run an experiment, then you use the results to determine if your guess was wrong. If it's wrong - then you refine your guess and repeat.
Letting data lead you results in the failed models we see so often, and is quite intellectually lazy. It takes more work and experience to do it the right way, but you end up with the proper answer
You can't 'see under the hood' so why bother? (Score:2)
'AI Is Still Not Ready' (Score:2)
not enough domain expertise (Score:2)
That seems unfortunately very common. Experts in a buzzword field (e.g. AI) slap their expertise on a field or product they have little expertise in and call it a day. After all, the buzzword does all the magic and it's usually good enough to attract investors/grants.
The opposite is very common too where buzzwords get slapped on otherwise functional products retroactively, e.g. all that IoT junk with abysmal security or "AI" products where the AI is a glorified if-then statement.
Buzzwords ruin everything.
Waiting for AI? (Score:1)
Not ready to predict it how WE want (Score:1)
A lead finder, not a prover (Score:1)
If they are accurate enough to say with a reasonably high probability somebody has the Covid, then followup tests can be done with slower but more accurate techniques.
AI is not ready for anything (Score:2)
really? (Score:1)