Paper Mills Have Flooded Science With 400,000 Fake Studies, Experts Warn (nature.com) 57
A group of scientific integrity experts is calling for urgent action to combat "paper mills" -- companies that sell fraudulent research papers and fake peer reviews. In a Nature comment piece published January 27, the experts warn that at least 400,000 papers published between 2000 and 2022 show signs of being produced by paper mills, while only 55,000 were retracted or corrected during that period.
The ghost of one million farts drowned in bleach (Score:2)
Goodhart's law (Score:5, Insightful)
Goodhart's law (https://en.wikipedia.org/wiki/Goodhart%27s_law) is "When a metric becomes a target, it ceases to function as a metric". People noticed that good academics tended to publish more than bad ones (though this is not a universal rule!) so we got targets for numbers of publications. And what's the result? Massive amounts of useless/marginal papers contributing only noise. Paper mills exist because bad actors want to compete and can win on these metrics. Hell some people are apparently able to produce multiple papers per day!
Want to fix this? Fix your metrics. Bad metrics destroy progress because they incentivize the wrong things. You ask for lots of papers, you get lots of noise. You ask for lots of citations, you get citation circles, huge self-citations and friend/courtesty citations, and reviewers demanding you cite them. You want "high impact" papers? You get Wakefield's high impact MMR work, researchers writing about hype instead of rigorous science. And they'll do it because we select out those who don't play the game. When your five good papers are outweighed in job and grant applications by twenty poor ones, that's your career ended in favour of the cheaters.
Anyone involved in coding knows this - you give a "lines of code" metric, you get lots of verbose code. You give a "commits" metric, you'll be innundated with pointless requests.
So, fix your metrics. Or better yet, get rid of them entirely and do the hard work of actually evaluating output properly.
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
Need number of papers with research grant funding (Score:3)
We need a statistic of how many of the 400,000 estimated research papers with fake data were funded, in whole or part, by government grants, grants from nonprofits, etc..
Next, for authors found to falsify a paper's data, we'd need statistics on how many other papers they've published along with grant dollars spent.
Finally, universities and nonprofits should be required to pay back some portion of a grant that resulted in published papers proven to have false data.
University of California
Frequently Asked Qu
Re: (Score:2)
Re: (Score:2)
Also no "peer review" can establish a paper's worth. To do that would require doing a good proportion of the work that went into the paper, and having knowledge that perhaps the "expert reviewers" lack.
Re: (Score:2)
Re: (Score:2)
A friend who knew a good deal about science and management told me that most really good scientists make at most three or four major discoveries in their careers. Some fewer. Obviously asking for 30 or more papers per years is absurd.
Quality should be dealt with at the higher level - the college or institute should set appropriate goals and then create teams of appropriate size and membership to tackle them. Perhaps the institutions should have to compete, rather than the individuals.
Re:Goodhart's law (Score:5, Insightful)
The fix is to incentivize critiques and analysis of the papers, and do that several layers deep, including meta moderation, targeting the critiques and analysis for review as well.
THIS is the basis for science and digging for truth is hard, tedious and takes a lot of time. It isn't perfect.
Re: (Score:2)
Re: (Score:1)
Absolutely no academic or researcher ever got a promotion based on the quality of their reviews.
Which is the problem that GP's suggesting we fix.
Re: (Score:2)
Another part of the fix is to stop using self-important, ugly, garbage words like "incentivize," and go back to using words like "encourage," which say what you mean without trying to impress your readers with how smart you are.
Re: (Score:2)
The fix is to incentivize critiques and analysis of the papers, and do that several layers deep, including meta moderation, targeting the critiques and analysis for review as well.
THIS is the basis for science and digging for truth is hard, tedious and takes a lot of time. It isn't perfect.
Also known as the /. Model
Re: (Score:1)
This.
You get what you measure.
I'd start reform by insisting that every paper include a necessary and sufficient falsifiable hypothesis statement. That will filter out 90% of the fraudulent/irreproducible trash. From both paper mills and "respectable" journals.
Re: (Score:1)
You think general relativity isn't falsifiable?
The first three tests, proposed by Albert Einstein in 1915, concerned the "anomalous" precession of the perihelion of Mercury, the bending of light in gravitational fields, and the gravitational redshift.
Einstein was a Popperian even before Popper wrote so eloquently on falsification fixing the demarcation problem. Einstein knew how to do science.
Re: (Score:2)
Too many submissions, not enough people to evaluate output properly.
Re:Goodhart's law (Score:4)
fix your metrics
I hate hate HATE the article count and citation count as metrics of research success. Not only does it incentivize quantity and linking your work to research clickbait, it promotes the myth of the 'independent researcher working alone in vacuum.' In my career I've always been more successful when I work in a department or setting that is active - lots of data to analyze, lots of money flowing around, etc. I'm the same person, but my output is so heavily dependent on those around me and the structures and resources available in that setting. The third problem with these metrics is there is no incentive to actually solve anything, as writing about the same problem endlessly (without making any real progress) is rewarded just the same as making actual progress.
I propose that the true measure of success should be at the team or department level (rather than individual), and should emphasize progress in addressing or solving an important problem. And rewards and improvements should be focused on the team and department level too (although sometimes you also have individuals in the team that need to develop new skills or change their behaviors). But that kind of analysis of success is much more difficult and time consuming than simply counting papers and citations and stupid impact factors...so no one ever does it.
Re: (Score:2)
I think the difficulty is genuine to some extent, and arises because there are so many scientists nowadays - most of them just plain honest people trying to make a living and perhaps get promotion or recognition.
Ironically, I think the system is broken because none of the higher-ups have given any useful thought to how science should be organised. As you point out, generating tens of thousands of more or less meaningless papers is not useful. What's needed is some way of setting important goals and organisi
Re: (Score:3)
Goodhart's Law really is the heart of the matter.
Please notice, also, that most of the foundations of almost every serious science were laid by amateurs - in the sense that they were people working for the satisfaction of it, not salaried employees.
Aristotle, Euclid, Archimedes, Leonardo da Vinci, Roger Bacon, Francis Bacon, Tycho Brahe, Kepler, Galileo, Descartes, Newton, Leibniz, Faraday, Babbage, Darwin, and on and on up to about WW2. None of them had "qualifications" - much. Some of them, being so incli
Well. It's time. (Score:4, Funny)
All papers from this point forward should be hand-written in the lead's handwriting, accompanied by a video of the writing process. And the pen(s), for forensic ink comparison.
Also, any respectable journal will not publish without suspending the submitter over a pit of spikes for two weeks while the work is authenticated. A fail gets them a quick drop.
Re:Well. It's time. (Score:5, Funny)
All papers from this point forward should be hand-written in the lead's handwriting, accompanied by a video of the writing process. And the pen(s), for forensic ink comparison.
Also, any respectable journal will not publish without suspending the submitter over a pit of spikes for two weeks while the work is authenticated. A fail gets them a quick drop.
A suspicious rise in the number of scientific studies being submitted from BDSM club members compared to non BDSM club members has appeared since the new rules were invoked. Studies are being conducted by the dom-sub curious to see why this particular club membership has had such a steep rise in scientific studies being produced.
"Paper mills" (Score:3, Funny)
Re: (Score:3)
Re: (Score:2)
Is this some sort of attack from rising fascism to disrupt and discredit science?
It's the same thing as anything...it is all about money. People/businesses with financial incentives to produce friendly "results" are going to put as many friendly "results" into existence as they can to justify what they are selling. If the results are valid, then great. If they are not valid, well, pretend they are anyways.
It is nothing new, as propaganda through media is as old as media.
so push academic dishonesty into the criminal cour (Score:2, Insightful)
so push academic dishonesty into the criminal court system?
That will really slow stuff down and can lead to judges / jury's getting the power to award degrees / class credits.
also can lead to Plagiarism tools going under COURT review with them forced to give out internal info to both party.
Also will an jury really send someone away and take the there guns rights over BS like self Plagiarism? Plagiarism due to not siting something the right way?
Re: (Score:1)
Re: (Score:2)
I don't think charging bad authors with fraud would work.
In the first place, in order to charge someone with fraud, you'd need to have a law in place that specifies the kind of fraud someone could be charged with. Setting aside possible 1A implications, I don't see how such a law could be constructed without it having a chilling effect on scientific discourse. Sometimes scientists extrapolate from known facts in ways that may or may not be confirmed later.
Second, the courts are the wrong place to review sci
Re: (Score:3)
It is possible to charge committing scientific misconduct with fraud using the fraud laws. In most cases, scientists are paid to do their work; and even if not, there is a public interest defence.
It is just a very blunt tool. Loss of professional accreditation, where it exists, and professional reputation is a rather sharper implement and is much more common.
I think all of these would be more effective if it wasn't for the pressure that the numerous measures and metrics apply.
Using fraud laws - who do you charge? (Score:2)
Re: (Score:3)
Re: (Score:2)
A better solution to academic dishonesty is for research funders to provide the working papers repository and require its use. Every raw image, every hypothesis discussed, lab notes, system configurations, etc should all be there. The daily activities of research should be fully visible to the funding organization. In this era it is horrifically wrong that this is not occurring. We lose so much and make cheating so much ea
Re: (Score:3)
AI isn't necessary to write a bogus research paper. Though it certainly helps.
It would be interesting (Score:2)
It would be interesting if one or more of these fraudulent studies by chance turned out to contain a major breakthrough in the field it was written about.
Re: (Score:2)
Re: (Score:3)
With my infinites, sure. However, a study was done on the Infinite Monkey theory if given practical constraints.
It turns out the observable universe is too finite to hold enough monkeys to get the job done within the universe's remaining lifetime.
So that is a bit of a bummer...
Re: (Score:2)
It turns out the observable universe is too finite to hold enough monkeys to get the job done within the universe's remaining lifetime.
What about statistical monkeys that learn word frequencies so well they can predict the next-word from the 500-billion parameter word distribution....
The Infinite Monkey Theorem [wikipedia.org] assumes the monkeys type randomly. That's why monkeys are used in the hypothetical problem.
Smarten-up the monkeys to know conditional word-frequencies and, sure, you might get a better chance of producing Hamlet, but without doing the numbers, I'd guess the probability is still vanishingly small.
Re: (Score:1)
Or a babbling orangutan who wins the Presidency via mass Gish Galloping.
Colleges expect publication (Score:2)
Who cares? (Score:3)
Something doesn't add up (Score:2)
Before relying on a research paper for anything important, wouldn't you first check with the source, or at least with a reputable verification agency?
If they just use Google to find papers without vetting, they are Grade A Morons.
If they are mostly used by trolls looking to bolster their case with fake studies, then there is probably no profit for the fake study maker. Unless their agenda is propaganda to give trolls scientific "legitimacy", such as "Transgender people are 40% more likely to eat pets".
In ot
Re: (Score:1)
Similar (Score:2)
A Paper Mill actually makes paper (Score:1)
Apparently you do not know this, but:
A Paper Mill is a manufacturing plant that physically makes paper comma you cloying headline-twisting jerk
A Paper Mill takes tree material and outputs blank physical paper for printing onto
A Paper Mill smells bad and all the cars in the parking lot are covered in a fine but harmless dust from the process
A Paper Mill is not a place where papers are written. Please choose less stupid, overlapping and confusing terms to make up and win likes
Their definition of "Fake" is broken. (Score:2)
-- Real data, faked peer review: FAKE
The rich have been buying fake studies for decades (Score:2)