FDA's New Drug Approval AI Is Generating Fake Studies (gizmodo.com) 40
An anonymous reader quotes a report from Gizmodo: Robert F. Kennedy Jr., the Secretary of Health and Human Services, has made a big push to get agencies like the Food and Drug Administration to use generative artificial intelligence tools. In fact, Kennedy recently told Tucker Carlson that AI will soon be used to approve new drugs "very, very quickly." But a new report from CNN confirms all our worst fears. Elsa, the FDA's AI tool, is spitting out fake studies.
CNN spoke with six current and former employees at the FDA, three of whom have used Elsa for work that they described as helpful, like creating meeting notes and summaries. But three of those FDA employees told CNN (paywalled) that Elsa just makes up nonexistent studies, something commonly referred to in AI as "hallucinating." The AI will also misrepresent research, according to these employees. "Anything that you don't have time to double-check is unreliable. It hallucinates confidently," one unnamed FDA employee told CNN. [...] Kennedy's Make America Healthy Again (MAHA) commission issued a report back in May that was later found to be filled with citations for fake studies. An analysis from the nonprofit news outlet NOTUS found that at least seven studies cited didn't even exist, with many more misrepresenting what was actually said in a given study. We still don't know if the commission used Elsa to generate that report.
FDA Commissioner Marty Makary initially deployed Elsa across the agency on June 2, and an internal slide leaked to Gizmodo bragged that the system was "cost-effective," only costing $12,000 in its first week. Makary said that Elsa was "ahead of schedule and under budget" when he first announced the AI rollout. But it seems like you get what you pay for. If you don't care about the accuracy of your work, Elsa sounds like a great tool for allowing you to get slop out the door faster, generating garbage studies that could potentially have real consequences for public health in the U.S. CNN notes that if an FDA employee asks Elsa to generate a one-paragraph summary of a 20-page paper on a new drug, there's no simple way to know if that summary is accurate. And even if the summary is more or less accurate, what if there's something within that 20-page report that would be a big red flag for any human with expertise? The only way to know for sure if something was missed or if the summary is accurate is to actually read the report. The FDA employees who spoke with CNN said they tested Elsa by asking basic questions like how many drugs of a certain class have been approved for children. Elsa confidently gave wrong answers, and while it apparently apologized when it was corrected, a robot being "sorry" doesn't really fix anything.
CNN spoke with six current and former employees at the FDA, three of whom have used Elsa for work that they described as helpful, like creating meeting notes and summaries. But three of those FDA employees told CNN (paywalled) that Elsa just makes up nonexistent studies, something commonly referred to in AI as "hallucinating." The AI will also misrepresent research, according to these employees. "Anything that you don't have time to double-check is unreliable. It hallucinates confidently," one unnamed FDA employee told CNN. [...] Kennedy's Make America Healthy Again (MAHA) commission issued a report back in May that was later found to be filled with citations for fake studies. An analysis from the nonprofit news outlet NOTUS found that at least seven studies cited didn't even exist, with many more misrepresenting what was actually said in a given study. We still don't know if the commission used Elsa to generate that report.
FDA Commissioner Marty Makary initially deployed Elsa across the agency on June 2, and an internal slide leaked to Gizmodo bragged that the system was "cost-effective," only costing $12,000 in its first week. Makary said that Elsa was "ahead of schedule and under budget" when he first announced the AI rollout. But it seems like you get what you pay for. If you don't care about the accuracy of your work, Elsa sounds like a great tool for allowing you to get slop out the door faster, generating garbage studies that could potentially have real consequences for public health in the U.S. CNN notes that if an FDA employee asks Elsa to generate a one-paragraph summary of a 20-page paper on a new drug, there's no simple way to know if that summary is accurate. And even if the summary is more or less accurate, what if there's something within that 20-page report that would be a big red flag for any human with expertise? The only way to know for sure if something was missed or if the summary is accurate is to actually read the report. The FDA employees who spoke with CNN said they tested Elsa by asking basic questions like how many drugs of a certain class have been approved for children. Elsa confidently gave wrong answers, and while it apparently apologized when it was corrected, a robot being "sorry" doesn't really fix anything.
It hallucinates confidently, (Score:5, Insightful)
Pretty much sums up the problem with all current LLMs. They aren't only wrong, the answers are phrased so definitively that they sound correct even when they aren't.
Re: It hallucinates confidently, (Score:1)
Are you envious?
Re: It hallucinates confidently, (Score:2)
Re: (Score:2)
Re: (Score:2)
I think it's because LLM "personalities" are biased by the owners / builders of it. They lack humility and self-reflection and thus don't process an answer at least twice before giving it.
hallucinates confidently (Score:5, Insightful)
Elsa just makes up nonexistent studies, something commonly referred to in AI as "hallucinating." The AI will also misrepresent research, according to these employees. "Anything that you don't have time to double-check is unreliable. It hallucinates confidently,"
And... Else gets nominated for a White House cabinet position in 3... 2... 1... /s
Re: hallucinates confidently (Score:3)
They just need to figure out how to make it fuck children and it can be president.
Re: (Score:3)
They just need to figure out how to make it fuck children and it can be president.
If it can convince them or their parents to skip getting vaccinated and avoid fluoride, then it could, at least, fuck them over...?
Re: (Score:2)
Thinking about this more, AI trained on a corpus of internet bullshit is going to be more likely to bullshit you about things that aren't convenient to fact check, because the legions of dipshits do exactly the same thing.
Still better than RFK (Score:4, Insightful)
Re: (Score:3)
I'd still trust the AI to make more sound decisions than RFK Jr. and it's probably a better person than him too.
Hannibal Lector would also fit both criteria. He's a great guy, apparently, maybe they should hire him. After all, despite all his little faults, no-one ever accused Dr. Lector of being bad at medicine. Or cooking.
Re: (Score:2)
A coin toss would be better at making critical decisions than that idiot. At least it would be correct sometimes.
Idiot (Score:5, Insightful)
So the anti-vaxxer is gung-ho for using the proven incredibly fallible new tech to approve new meds, but shits on decades old proven safe vaccinations? What a fucking idiot.
Re: (Score:1, Interesting)
A: President donald "the kiddy diddler" trump. Fact: trump fucks children
Re:Idiot (Score:5, Informative)
I know you're just a fucking idiot, but for anyone who's interested and didn't see it at the time, there are two important things to note about vaccine safety:
1. Side effects happen really fast because they're normally from the immune system rather than the vaccine itself, which is broken down really quickly by the body, and adaptive immunity builds by itself over days to weeks and the body tightly regulates it.
2. What matters with trials is statistical power, ie the size of the sample. Covid vaccine trials had a n of 40k, which is 20x a typical trial. And they recruited much faster, too (trials typically take years to recruit, but obviously: the pandemic affected everyone while not everyone has, say, the pulmonary arterial hypertension needed to test a PAH drug; and there were a lot of people feeling pretty damn motivated to participate in trials and get a vaccine out given the impact of the pandemic)
Re: (Score:2)
Yes. This was done very well under time pressure. But the anti-vaxx idiots lack working minds. They cannot distinguish between fact and hallucination. The funny thing is that even the piece-of-shit that started the movement was not anti-vaxx but just sought to discredit a competing vaccine. I bet most of the anti-vaxx cretins do not even know that.
Re: Idiot (Score:1)
Are you the technician from the Deep Learning South Park episode? Or gweihir?
"School counselor Mr. Mackey informs Stan's class that a student used OpenAI technology for schoolwork. A "technician" dressed as a falconer arrives with his falcon Shadowbane to analyze the students' work and identify the cheater."
Re:Idiot (Score:4)
Don't try and wrap your head around MAGA logic. Today they're yelling something about Obama while Mike Johnson shut down the house early to avoid voting on the Epstein files.
Re:Idiot (Score:5, Informative)
Mike Johnson shut down the house early to avoid voting on the Epstein files.
He didn't send them home early enough. GOP-led House panel votes to subpoena Epstein files [axios.com]
A Republican-led House subcommittee on Wednesday passed a Democrat's motion to subpoena the Justice Department's documents on Jeffrey Epstein.
The panel voted 8 -2 in favor of the subpoena, with only Chair Clay Higgins (R-LA) and Rep. Andy Biggs (R-AZ) voting against it.
Reps. Nancy Mace (R-SC), Scott Perry (R-PA) and Brian Jack (R-GA) voted for the subpoena along with the five Democrats on the panel.
Several right-wing House Republicans, including Reps. Lauren Boebert (R-CO) and Paul Gosar (R-AZ), were absent from the vote.
Re: (Score:1)
It will never not be funny if it turns out that the MAGAts' one actually true conspiracy is that a rich and powerful pedo used shadowy criminal associations to murder Epstein to prevent himself being outed as such... and it was the orange idiot who did it. And it will be even funnier because it's modestly plausible too. He's certainly squirming round in humiliation rn
Re: Idiot (Score:1)
Trump couldn't do it... Alone anyway. There had to be a conspiracy involved to protect the conspiracy.
maggots with modpoints (Score:2)
How does it feel to make hating pedos your entire identity and then find out you voted for one, maggots?
Re: (Score:3)
He is truly the American Jimmy Saville. It was all obvious from the outset. He unabashedly walked in on 15 year old girls as they changed for his disgusting beauty pageants, FGS!
Re: (Score:2)
It's always disgusting to see people come to his defense. It's heartbreaking when it happens here. This is supposed to be a place for people who care about knowing things. And indeed I assume that the people defending him actually know he's a child fucker, so that means that a significant percentage of the people here who regularly get modpoints are in favor of child rape.
Re: (Score:2)
Yep. What a complete failure as a person. But Trump likes these, because if he surrounds himself with these then he does not look like the clueless criminal moron he is. And the MAGAs are too dumb to recognize extreme stupid anyways. In fact it makes them probably feel right at home and comfortable.
All part of the plan. (Score:5, Insightful)
Having science polluted by AI hallucinations seems an easy way to increase doubt in science, and allow q-anon level conspiracies to appear like a more valid alternate viewpoint.
Re: (Score:2)
And at the same time, with a serious face, used to support whatever their position is. Accompanied by the usual stance of saying every other position is a lie.
Re: hallucinated people (Score:1)
You need to work on your LLM, it's not good at producing English.
In today's meeting (Score:4, Insightful)
Executive: We need AI!
Me: Why? Are we out of RI?
Executive: RI? What's that?
Me: Real Intelligence
How do we trust these things? (Score:3)
We know it makes stuff up, like studies, documentation...
How are meeting notes from this thing trustworthy in any way?
Re: (Score:2)
I Don't Get The Enthusiasm For AI Systems (Score:2)
I mean, just the other day:
Vibe Coding Goes Wrong As AI Wipes Entire Database
https://hackaday.com/2025/07/2... [hackaday.com]
Re: (Score:2)
Yep, same here. You have to be blind and dumb to think these are good tools. I take it as Yet Another Proof that the average person is pretty dumb and cannot fact-check for shit.
More Trump-Brand Idiocracy (Score:2)
Perfect! (Score:2)
"AI" aka "automated bullshit generator" is 100% perfect for this bullshit administration.
Works as expected (Score:2)
As expected by anybody with an IQ above room temperature, that is. Obviously the MAGAs lack that.