How Research Credibility Suffers in a Quantified Society (socialsciencespace.com) 32
An anonymous reader shares a report: Academia is in a credibility crisis. A record-breaking 10,000 scientific papers were retracted in 2023 because of scientific misconduct, and academic journals are overwhelmed by AI-generated images, data, and texts. To understand the roots of this problem, we must look at the role of metrics in evaluating the academic performance of individuals and institutions.
To gauge research quality, we count papers, citations, and calculate impact factors. The higher the scores, the better. Academic performance is often expressed in numbers. Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality. Besides, numbers seem objective and fair, which is why we use them to allocate status, tenure, attention, and funding to those who score well on these indicators.
The result of this? Quantity is often valued over quality. In The Quantified Society I coin the term "indicatorism": a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters. It seems we're sometimes busier with "scoring" and "producing" than with "understanding." As a result, some started gaming the system. The rector of one of the world's oldest universities, for one, set up citation cartels to boost his citation scores, while others reportedly buy(!) bogus citations. Even top-ranked institutions seem to play the indicator game by submitting false data to improve their position on university rankings!
To gauge research quality, we count papers, citations, and calculate impact factors. The higher the scores, the better. Academic performance is often expressed in numbers. Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality. Besides, numbers seem objective and fair, which is why we use them to allocate status, tenure, attention, and funding to those who score well on these indicators.
The result of this? Quantity is often valued over quality. In The Quantified Society I coin the term "indicatorism": a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters. It seems we're sometimes busier with "scoring" and "producing" than with "understanding." As a result, some started gaming the system. The rector of one of the world's oldest universities, for one, set up citation cartels to boost his citation scores, while others reportedly buy(!) bogus citations. Even top-ranked institutions seem to play the indicator game by submitting false data to improve their position on university rankings!
Where there is incentive, there is evil. (Score:2, Insightful)
Nothing about being a scientist makes someone into an honest person.
Researchers need to show results in order to continue being researchers, not to mention put bread on the table. There is fame to be gained as well. AND, science has the ability to motivate action on a large scale, which means that corrupt political actors have very strong incentive to (one way or another) intervene and insure that the results are what they want them to be.
Science is valuable enough to be worth protecting. But that doesn'
Re: (Score:3, Insightful)
AND, science has the ability to motivate action on a large scale, which means that corrupt political actors have very strong incentive to (one way or another) intervene and insure that the results are what they want them to be.
OR, alternatively, those corrupt political actors can attempt to discredit science itself because it doesn't support their overall agenda.
Re: (Score:2)
I'm not even sure if it really matters at
Re: (Score:2)
It really doesn't matter what anyone says or does because the universe is the ultimate arbiter and those who seek to understand the truth of how it operates will be more successful over the long term than those who try to suppress that truth.
There is little indication that humans (the only ones currently able to "seek to understand the truth") will outlive microbes or ants on this planet. And even among humans, those with irrational beliefs in absolutely not fact-based religions telling them to reproduce, regardless of circumstances, proliferate at much higher rates than the ones occupied with doing science stuff.
Humans only Hope to Outlive Sun (Score:2)
There is little indication that humans (the only ones currently able to "seek to understand the truth") will outlive microbes or ants on this planet.
Perhaps not "on this planet" but this planet has a finite time for which it can support life. If any life on Earth, microbe or otherwise wants to outlive the sun, the only hope there is for it currently is humans.
Re: (Score:2, Insightful)
I think that's right, to an extent, but scientists have been able to "put bread on the table" in the past without corruption or playing a game. The question is what's different?
You know what else has changed? The idea that corporations are responsible only for generating profit for shareholders. Could these issues be related?
We don't have to accept that corruption exists everywhere, that it always has and always will. We can ask why we have the problem now. The solution may not be "easy to do" but may
Re: (Score:2)
I think that's right, to an extent, but scientists have been able to "put bread on the table" in the past without corruption or playing a game. The question is what's different?
There are more of them, and the competition is fiercer. The highest-impact publications are largely the same but there are many times more researches trying to their papers published. Papers that would have been published in relatively high-impact journals 20-30 years ago are not even sent to peer review for less glamorous ones today.
Also, one's success - as a researcher or as a university - is quantified more stringently than it used to.
And it doesn't help if your competitors are willing to resort to any
"The Big Crunch" is why there is much incentive (Score:5, Informative)
From 1994 by Dr. David Goodstein, then vice-provost of Caltech: https://www.its.caltech.edu/~d... [caltech.edu] ... ..."
"The period 1950-1970 was a true golden age for American science. Young Ph.D's could choose among excellent jobs, and anyone with a decent scientific idea could be sure of getting funds to pursue it. The impressive successes of scientific projects during the Second World War had paved the way for the federal government to assume responsibility for the support of basic research. Moreover, much of the rest of the world was still crippled by the after-effects of the war. At the same time, the G.I. Bill of Rights sent a whole generation back to college transforming the United States from a nation of elite higher education to a nation of mass higher education. Before the war, about 8% of Americans went to college, a figure comparable to that in France or England. By now more than half of all Americans receive some sort of post-secondary education. The American academic enterprise grew explosively, especially in science and technology. The expanding academic world in 1950-1970 created posts for the exploding number of new science Ph.D.s, whose research led to the founding of journals, to the acquisition of prizes and awards, and to increases in every other measure of the size and quality of science. At the same time, great American corporations such as AT&T, IBM and others decided they needed to create or expand their central research laboratories to solve technological problems, and also to pursue basic research that would provide ideas for future developments. And the federal government itself established a network of excellent national laboratories that also became the source of jobs and opportunities for aspiring scientists. Even so, that explosive growth was merely a seamless continuation of a hundred years of exponential growth of American science. It seemed to one and all (with the notable exception of Derek da Solla Price) that these happy conditions would go on forever.
By now, in the 1990's, the situation has changed dramatically. With the Cold War over, National Security is rapidly losing its appeal as a means of generating support for scientific research. There are those who argue that research is essential for our economic future, but the managers of the economy know better. The great corporations have decided that central research laboratories were not such a good idea after all. Many of the national laboratories have lost their missions and have not found new ones. The economy has gradually transformed from manufacturing to service, and service industries like banking and insurance don't support much scientific research. To make matters worse, the country is almost 5 trillion dollars in debt, and scientific research is among the few items of discretionary spending left in the national budget. There is much wringing of hands about impending shortages of trained scientific talent to ensure the Nation's future competitiveness, especially since by now other countries have been restored to economic and scientific vigor, but in fact, jobs are scarce for recent graduates. Finally, it should be clear by now that with more than half the kids in America already going to college, academic expansion is finished forever.
The crises that face science are not limited to jobs and research funds. Those are bad enough, but they are just the beginning. Under stress from those problems, other parts of the scientific enterprise have started showing signs of distress. One of the most essential is the matter of honesty and ethical behavior among scientists.
The public and the scientific community have both been shocked in recent years by an increasing number of cases of fraud committed by scientists. There is little doubt that the perpetrators in these cases felt themselves under intense pressure to compete for scarce resources, even by cheating if necessary. As the pressure increases, this kind of dishonesty is almost sure to become more common.
Goodhart's law [Re:Where there is incentive, t...] (Score:4, Insightful)
An example of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure".
https://quickonomics.com/terms... [quickonomics.com]
Re: (Score:2)
Reducing complexity makes it easier to cheat.
The coming crapflood apocalypse. (Score:2)
Glad to see the scientific journals won't be immune to the coming crapflood apocalypse that LLM driven AI is creating for most of the world. When the AIs start feeding on their own filth and still spitting out content faster than anyone else can, will we be smart enough to just shut the whole thing down? Or will we just let the entire information sphere disappear into the filth we're flooding it with?
The future is brutal.
Re: (Score:2)
The future is brutal.
"The future's so bright, I gotta wear shades." --Timbuk 3, 1986. Originally intended as snark against Reagan, I think it applies equally well to any dystopian hell. And I think you're right, I think we're headed to one. Or, have been in one for a while already. 2020 was just the tip of the spear.
What about all the unreplicated papers? (Score:1, Troll)
Those papers get reported her and in the mainstream media and, worse, get used by psychologists to pretend their voodoo is based on science, and PEOPLE SUFFER SEVERELY BECAUSE OF IT.
Re: (Score:3)
Haven't you heard? Suffering is fine so long as someone, somewhere, makes a profit off of it.
Re: (Score:2)
Re: (Score:2)
And now that scammers like "betterhelp" can bring everyone into the job of being a "therapist" for others, you can profit, too, by just pretending you are a "professional" and talking to gullible people on the phone. Which brings profit also to the Influencers, who lured the gullible people into subscribing for "betterhelp".
And now that scammers like "betterhelp" can bring everyone into the job of being a "therapist" for others, you can profit, too, by just pretending you are a "professional" and talking to gullible people on the phone. Which brings profit also to the Influencers, who lured the gullible people into subscribing for "betterhelp".
Wasn't betterhelp caught collecting data from their "Patients" and selling it on? Places like that seem like the perfect storm of the modern age, collect the data, scam the plebes, sell their deepest secrets.
Re: (Score:3)
Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. https://science.slashdot.org/s... [slashdot.org]
On almost every subject, there are competing theories (sometimes mutually exclusive, they couldn't both be true). It is nearly unavoidable to not have biases, as part of the scientific method is generating a hypothesis then proving or disproving it. This often shows up in determining root causes of diseases. Another problem I often see is a claim will be made in a paper and a citation to back it up, but looking into the citation, it does
Re: (Score:1)
Physics does not have a replication problem.
Not a new thing (Score:2, Informative)
In the IT world, for example, Joel Splonksy (of Fog Creek Software fame) has been writing about measurement dysfunction and gamification since last century.
Metrics drive behavior so be careful what you measure.
Re:Not a new thing (Score:4, Insightful)
As XKCD said about Goodhart's Law [explainxkcd.com], "When a metric becomes a target, it cease to be a good metric."
Massive punishment for blatant perpetrators (Score:2)
You create bonus citations? You lose all your degrees, your job etc...
Of course part of the problem is that lots of crappy papers have more impact than a few classics. Higgs, of the Boson fame, reckons that he wouldn't get the chance these days to do the deep thinking that led him to hypothesise the Boson's existence.
Too many people are doing PhDs, which are a nice little earner for universities, but have no sensible long term career path.
Re: (Score:2)
You create bonus citations? You lose all your degrees, your job etc...
That's the same simplistic mindset that got us in the mess in the first place. Reduce complexity, get a simple number, and then act with force on it.
What if your department head was trying to get a higher score on some dubious ranking and thus bumped up citations? Why should you lose your degree? What if it was the dean of the whole institution, trying to position it better for getting grants? What if you were one of several contributors to a paper (most papers are collaborations anyway), and some of your
standard BS (Score:5, Insightful)
"Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality."
Just the standard bullshit. Par for the course.
First off, "quantification" just means measuring stuff, and measuring stuff does NOT "reduce complexity" nor does it make anything "manageable". Sure it allows easy comparisons of what is measured, but value is only provided if the right things are measured, and measured correctly. Finally, it provides bureaucrats with false comfort and control. That may be true, but also part of the problem.
"The result of this? Quantity is often valued over quality. "
Wrong once again. "Quantification" is not a process of producing quantity, it's is the process of measuring. Quantification does NOT value quantity of quality. We need to question how ignorant the author is.
"The Quantified Society I coin the term "indicatorism"..."
The term "quantified society", if it exists at all, would not mean what the author thinks. Science is predicated on "quantifying", yet the author thinks it is a term of derision.
"...a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters."
And what really matters? Surely it cannot be precision in language. I get the author's intent, but he's guilty of precisely the same laziness he criticizes.
"As a result, some started gaming the system."
This isn't a problem with Academia, it's a problem with every aspect of modern society. Everything is to be gamed, right SuperKendall?
And it should be understood that blaming metrics is not a solution, that is unless you are MAGA. The attempt at objective measurability is not at fault, it is laziness and corruption that is. We have an integrity problem, not an "indicatorism" problem. In fact, I'd say the author of this article demonstrates the problem, not the solution.
Re: (Score:2)
A great example is how numbers going up and down are used to confuse people about economic policy.
It's going to be interesting to see what happens when grocery prices don't fall as promised.
Re: (Score:2)
It's going to be interesting to see what happens when grocery prices don't fall as promised.
My bet is that anyone who doesn't forget about it in 4 years, will think it was the fault of their political rivals.
Re: (Score:2)
The trouble isn't measuring things, it is the use of bad proxy measures. Much of the time the thing you want is really complex to the point it is described as "subjective", such as "which is the best college for me". And then someone wants to put a number to it, because numbers are easy and "objective". So for example some asshole decides low acceptance rate indicates a good college lots of people want to go to, when in reality its a measure of how many people guess wrong as to their odds of being accepted.
PhB PhD (Score:2)
Use a real metric like real managers do: lines of code/text
The whole system is wrong (Score:2)
Why not say what it really is.... (Score:3)
Most metrics are crap (Score:2)
And the more complex the subject, the worse it gets. Sure, metrics for simpler things are, with very careful evaluation and interpretation, quite useful. But that is about it. In addition, any metric can be gamed.
The metrics for "scientific productivity" are both applied to a very complex subject and very easy to game. Hence they are utter crap. Produce 20 crappy, marginally incremental, but nicely looking papers per year? Stellar researcher! Produce one paper that significantly advances the state-of-the-ar