Journals Infiltrated With 'Copycat' Papers That Can Be Written By AI (nature.com) 34
An analysis of a literature database finds that text-generating AI tools -- including ChatGPT and Gemini -- can be used to rewrite scientific papers and produce 'copycat' versions that are then passed off as new research. Nature: In a preprint posted on medRxiv on 12 September, researchers identified more than 400 such papers published in 112 journals over the past 4.5 years, and demonstrated that AI-generated biomedicine studies could evade publishers' anti-plagiarism checks.
The study's authors warn that individuals and paper mills -- companies that produce fake papers to order and sell authorships -- might be exploiting publicly available health data sets and using large language models (LLMs) to mass-produce low-quality papers that lack scientific value.
"If left unaddressed, this AI-based approach can be applied to all sorts of open-access databases, generating far more papers than anyone can imagine," says Csaba Szabo, a pharmacologist at the University of Fribourg in Switzerland, who was not involved in the work. "This could open up Pandora's box [and] the literature may be flooded with synthetic papers."
The study's authors warn that individuals and paper mills -- companies that produce fake papers to order and sell authorships -- might be exploiting publicly available health data sets and using large language models (LLMs) to mass-produce low-quality papers that lack scientific value.
"If left unaddressed, this AI-based approach can be applied to all sorts of open-access databases, generating far more papers than anyone can imagine," says Csaba Szabo, a pharmacologist at the University of Fribourg in Switzerland, who was not involved in the work. "This could open up Pandora's box [and] the literature may be flooded with synthetic papers."
Wait, (Score:2)
do they just allow anybody to submit a paper for review? That's Mistake #1, #2, and #3. Vet the sending org first.
Re: (Score:2)
Re: (Score:2)
People who were happy to purchase "Bob's Research Digest & Ocean Cruise Guide" from Mr. Trenchcoat probably won't care if AI stuffs it with yet more riff-raff "research" because they probably didn't care before.
Something tells me they are buying scientific-sounding "justifications" for something else dodgy rather than genuine knowledge. RFK Jr's group is an example candidate to claim Tums causes autism or whatnot. Real journals vet submitters.
#ProveMeWrong
Re: (Score:1)
do they just allow anybody to submit a paper for review?
I would hope that a non-credentialed "citizens scientist" could submit work to a journal, with the caveat that he would not be given the presumption of competence and presumption of integrity that a well-known professional scientist or someone employed as a scientist with a reputable institution or company would get.
In other words, if your no-established-reputation friendly neighborhood citizen-scientist did some high-quality, valuable research and submitted it to an on-topic journal, they should at least l
Re: (Score:1)
> I would hope that a non-credentialed "citizens scientist" could submit work to a journal...
It's fair to request a written endorsement from an established researcher first. Accepting papers from Nigerian Princes sight unseen has been a stupid idea since the first Nigerian Prince scam.
When accepting submissions from Nigerian ... (Score:1)
... princes, demand payment up front, charge at least 15 million United States Dollars, and require them to deliver payment in person to the offices of a three-letter-agency in Washington, D. C.
Re: (Score:1)
I evolved a more flexible Prince that makes catchy deals.
Re: (Score:1)
Only if you want to ensure that anything innovative gets filtered out and The Science becomes a circlejerk around The Established Narrative. Which the last fifty years of gatekeeping pretty much have.
Even longer, really, when you consider how long it took to convince surgeons that maybe washing their hands for surgery might be a good idea.
Yeah, there are a lot of nuts around. But there are also a lot of nuts and grifters inside The Science who don't want anyone spoiling their fun.
Re: (Score:2)
Re: (Score:2)
And yet Google Scholar won't allow me to verify my profile with my e-mail address from my university's alumni domain, because obviously freshers might do research but graduates can't.
Re: (Score:2)
These papers come from professional scientists in legitimate orgs. Figure 5 from the main paper https://www.medrxiv.org/conten... [medrxiv.org] shows several readable references. I picked two of such paper families (from different authors): https://pubmed.ncbi.nlm.nih.go... [nih.gov] https://pubmed.ncbi.nlm.nih.go... [nih.gov] They come from apparently legitimate doctors in hospitals in China and are published in Scientific Reports, a well known publications from the Nature group (not their best, but reputable). Hospital doctors are said
Re: (Score:2)
do they just allow anybody to submit a paper for review? That's Mistake #1, #2, and #3. Vet the sending org first.
Well, yes, and this is a good thing. Before the days of universal blind reviews, accepting and rejecting papers based on the reputation of authors used to happen, and that was a bad thing.
The solution is not to restrict submissions but rather to improve reviews.
Eventually this is less of a problem that it might appear to be. It might even be a good thing because it's the lesser conferences and journals that have fewer resources to devote to reviews. The top conferences have multiple reviewers that are ar
Authenticate the authors? (Score:1)
If this keeps up, the academic-journal industry will need to find a way to authenticate primary authors' identities and, for authors who don't have a proven track record, authenticate that the primary authors actually did the research behind the paper or (for lead authors on a large team) at least that they are familiar enough with the research to defend it and supervised the work closely enough that they can attest that it was done ethically.
Authors who can't be authenticated against a passport or similar
Please, dear god of Slashdot... (Score:1)
Re: Dupe? (Score:1)
Thanks to AI now slashdot will get mashup dupes that are amalgamated parts of articles already submitted. It will be fun the first few times...
AI can't do anything 'new'. (Score:2)
Using AI to create anything labeled new should be banned in most professions. The best AI can do is simple fact checking, where it flags questionable content for human review.
In all other instances current AI is only good for entertainment purposes.
NigerianPrinceGPT (Score:1)
Re: AI can't do new, only regurgitate patterns
It only has to be good enough to fool humans, not cure cancer.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
You are wrong. AI has done mathematical proofs that were new. It *can* only be original by combining existing information into new patterns, but if the "rules of inference" are good, this can allow it to create something new and good.
OTOH, you are partially correct, in that it can't derive anything that wasn't already implicitly implied by the existing knowledge.,,because it can't currently run its own experiments.
N.B.: This is a comment about "AI" not about pure "LLM"s. Pure LLMs are a lot less reliabl
Re: (Score:2)
Re: (Score:2)
Crisis averted (Score:3)
Publish or Perish (Score:3)
The problem is that Academics must publish X papers per year just to keep their jobs, X+Y if they want a chance for their career to move forward.
Are we surprised that an industry grew up with AI-ghostwriters producing papers to order, and low-quality journals publishing anything they are presented with (for a fee)?
That which is measured is improved. Nobody said they had to be GOOD papers -just that there must be papers published.
Fake authors (Score:2)
Re: (Score:2)
While it certainly could be taken up by an AI, it's a perfectly ordinary Hungarian name.
A human method for human works (Score:2)
A new validation mechanism is needed to verify and filter for human authored works in a large and growing variety of fields. This will likely involve being not lazy, and not relying on AI itself to vet for human created works. The current methods are obviously less and less usable as AI becomes more and more skilled at impersonating the tone and feel of human authored works.
I think this will necessarily mean a return to a more analog, labor intensive review of works and manual vetting of authors through
Re: (Score:2)
The particular problem here (copycat papers) is not difficult to solve. It only requires the publishers (at least the reputable ones) to start *requiring* an ORCID number for the authors, and asking the reviewers to check the incremental nature since the last published paper using the ORCID record. The current flaw is these authors above published without an ORCID so the reviewers don't have an immediate way to check previous papers, and didn't spend time googling for the previous papers.
Again, simple to solve. (Score:2)
Re: (Score:2)