How Research Credibility Suffers in a Quantified Society (socialsciencespace.com) 28
An anonymous reader shares a report: Academia is in a credibility crisis. A record-breaking 10,000 scientific papers were retracted in 2023 because of scientific misconduct, and academic journals are overwhelmed by AI-generated images, data, and texts. To understand the roots of this problem, we must look at the role of metrics in evaluating the academic performance of individuals and institutions.
To gauge research quality, we count papers, citations, and calculate impact factors. The higher the scores, the better. Academic performance is often expressed in numbers. Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality. Besides, numbers seem objective and fair, which is why we use them to allocate status, tenure, attention, and funding to those who score well on these indicators.
The result of this? Quantity is often valued over quality. In The Quantified Society I coin the term "indicatorism": a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters. It seems we're sometimes busier with "scoring" and "producing" than with "understanding." As a result, some started gaming the system. The rector of one of the world's oldest universities, for one, set up citation cartels to boost his citation scores, while others reportedly buy(!) bogus citations. Even top-ranked institutions seem to play the indicator game by submitting false data to improve their position on university rankings!
To gauge research quality, we count papers, citations, and calculate impact factors. The higher the scores, the better. Academic performance is often expressed in numbers. Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality. Besides, numbers seem objective and fair, which is why we use them to allocate status, tenure, attention, and funding to those who score well on these indicators.
The result of this? Quantity is often valued over quality. In The Quantified Society I coin the term "indicatorism": a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters. It seems we're sometimes busier with "scoring" and "producing" than with "understanding." As a result, some started gaming the system. The rector of one of the world's oldest universities, for one, set up citation cartels to boost his citation scores, while others reportedly buy(!) bogus citations. Even top-ranked institutions seem to play the indicator game by submitting false data to improve their position on university rankings!
Where there is incentive, there is evil. (Score:2, Insightful)
Nothing about being a scientist makes someone into an honest person.
Researchers need to show results in order to continue being researchers, not to mention put bread on the table. There is fame to be gained as well. AND, science has the ability to motivate action on a large scale, which means that corrupt political actors have very strong incentive to (one way or another) intervene and insure that the results are what they want them to be.
Science is valuable enough to be worth protecting. But that doesn'
Re:Where there is incentive, there is evil. (Score:4, Insightful)
AND, science has the ability to motivate action on a large scale, which means that corrupt political actors have very strong incentive to (one way or another) intervene and insure that the results are what they want them to be.
OR, alternatively, those corrupt political actors can attempt to discredit science itself because it doesn't support their overall agenda.
Re: (Score:2)
I'm not even sure if it really matters at
Re: (Score:2)
It really doesn't matter what anyone says or does because the universe is the ultimate arbiter and those who seek to understand the truth of how it operates will be more successful over the long term than those who try to suppress that truth.
There is little indication that humans (the only ones currently able to "seek to understand the truth") will outlive microbes or ants on this planet. And even among humans, those with irrational beliefs in absolutely not fact-based religions telling them to reproduce, regardless of circumstances, proliferate at much higher rates than the ones occupied with doing science stuff.
Humans only Hope to Outlive Sun (Score:2)
There is little indication that humans (the only ones currently able to "seek to understand the truth") will outlive microbes or ants on this planet.
Perhaps not "on this planet" but this planet has a finite time for which it can support life. If any life on Earth, microbe or otherwise wants to outlive the sun, the only hope there is for it currently is humans.
Re: (Score:3)
I think that's right, to an extent, but scientists have been able to "put bread on the table" in the past without corruption or playing a game. The question is what's different?
You know what else has changed? The idea that corporations are responsible only for generating profit for shareholders. Could these issues be related?
We don't have to accept that corruption exists everywhere, that it always has and always will. We can ask why we have the problem now. The solution may not be "easy to do" but may
Society (Score:1)
I think that's right, to an extent, but scientists have been able to "put bread on the table" in the past without corruption or playing a game. The question is what's different?
What is different is society. The UK had a PM who was convicted and fined for breaking a law his government passed and yet that did not result in an major scandal or the collapse of his government. The US is about to have a president convicted of a felony and yet his party seemed to have no problem with that. Could you imagine either of these things happening 50 years ago? I'm not sure whether corporate corrupt is another symptom or part of the cause but with societal standards of acceptable behaviour coll
Re: (Score:2)
I think that's right, to an extent, but scientists have been able to "put bread on the table" in the past without corruption or playing a game. The question is what's different?
There are more of them, and the competition is fiercer. The highest-impact publications are largely the same but there are many times more researches trying to their papers published. Papers that would have been published in relatively high-impact journals 20-30 years ago are not even sent to peer review for less glamorous ones today.
Also, one's success - as a researcher or as a university - is quantified more stringently than it used to.
And it doesn't help if your competitors are willing to resort to any
"The Big Crunch" is why there is much incentive (Score:3)
From 1994 by Dr. David Goodstein, then vice-provost of Caltech: https://www.its.caltech.edu/~d... [caltech.edu]
"The period 1950-1970 was a true golden age for American science. Young Ph.D's could choose among excellent jobs, and anyone with a decent scientific idea could be sure of getting funds to pursue it. The impressive successes of scientific projects during the Second World War had paved the way for the federal government to assume responsibility for the support of basic research. Moreover, much of the rest of the
Goodhart's law [Re:Where there is incentive, t...] (Score:3)
An example of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure".
https://quickonomics.com/terms... [quickonomics.com]
Re: (Score:2)
Reducing complexity makes it easier to cheat.
The coming crapflood apocalypse. (Score:2)
Glad to see the scientific journals won't be immune to the coming crapflood apocalypse that LLM driven AI is creating for most of the world. When the AIs start feeding on their own filth and still spitting out content faster than anyone else can, will we be smart enough to just shut the whole thing down? Or will we just let the entire information sphere disappear into the filth we're flooding it with?
The future is brutal.
Re: (Score:2)
The future is brutal.
"The future's so bright, I gotta wear shades." --Timbuk 3, 1986. Originally intended as snark against Reagan, I think it applies equally well to any dystopian hell. And I think you're right, I think we're headed to one. Or, have been in one for a while already. 2020 was just the tip of the spear.
Re: (Score:3)
Haven't you heard? Suffering is fine so long as someone, somewhere, makes a profit off of it.
Re: (Score:2)
Not a new thing (Score:1)
In the IT world, for example, Joel Splonksy (of Fog Creek Software fame) has been writing about measurement dysfunction and gamification since last century.
Metrics drive behavior so be careful what you measure.
Re: (Score:3)
As XKCD said about Goodhart's Law [explainxkcd.com], "When a metric becomes a target, it cease to be a good metric."
Massive punishment for blatant perpetrators (Score:2)
You create bonus citations? You lose all your degrees, your job etc...
Of course part of the problem is that lots of crappy papers have more impact than a few classics. Higgs, of the Boson fame, reckons that he wouldn't get the chance these days to do the deep thinking that led him to hypothesise the Boson's existence.
Too many people are doing PhDs, which are a nice little earner for universities, but have no sensible long term career path.
Re: (Score:2)
You create bonus citations? You lose all your degrees, your job etc...
That's the same simplistic mindset that got us in the mess in the first place. Reduce complexity, get a simple number, and then act with force on it.
What if your department head was trying to get a higher score on some dubious ranking and thus bumped up citations? Why should you lose your degree? What if it was the dean of the whole institution, trying to position it better for getting grants? What if you were one of several contributors to a paper (most papers are collaborations anyway), and some of your
standard BS (Score:5, Insightful)
"Why? Quantification reduces complexity, makes academia manageable, allows easy comparisons among scholars and institutions, and provides administrators with a feeling of grip on reality."
Just the standard bullshit. Par for the course.
First off, "quantification" just means measuring stuff, and measuring stuff does NOT "reduce complexity" nor does it make anything "manageable". Sure it allows easy comparisons of what is measured, but value is only provided if the right things are measured, and measured correctly. Finally, it provides bureaucrats with false comfort and control. That may be true, but also part of the problem.
"The result of this? Quantity is often valued over quality. "
Wrong once again. "Quantification" is not a process of producing quantity, it's is the process of measuring. Quantification does NOT value quantity of quality. We need to question how ignorant the author is.
"The Quantified Society I coin the term "indicatorism"..."
The term "quantified society", if it exists at all, would not mean what the author thinks. Science is predicated on "quantifying", yet the author thinks it is a term of derision.
"...a blind focus on enhancing indicators in spreadsheets, while losing sight of what really matters."
And what really matters? Surely it cannot be precision in language. I get the author's intent, but he's guilty of precisely the same laziness he criticizes.
"As a result, some started gaming the system."
This isn't a problem with Academia, it's a problem with every aspect of modern society. Everything is to be gamed, right SuperKendall?
And it should be understood that blaming metrics is not a solution, that is unless you are MAGA. The attempt at objective measurability is not at fault, it is laziness and corruption that is. We have an integrity problem, not an "indicatorism" problem. In fact, I'd say the author of this article demonstrates the problem, not the solution.
Re: (Score:2)
A great example is how numbers going up and down are used to confuse people about economic policy.
It's going to be interesting to see what happens when grocery prices don't fall as promised.
Re: (Score:2)
It's going to be interesting to see what happens when grocery prices don't fall as promised.
My bet is that anyone who doesn't forget about it in 4 years, will think it was the fault of their political rivals.
Re: (Score:2)
The trouble isn't measuring things, it is the use of bad proxy measures. Much of the time the thing you want is really complex to the point it is described as "subjective", such as "which is the best college for me". And then someone wants to put a number to it, because numbers are easy and "objective". So for example some asshole decides low acceptance rate indicates a good college lots of people want to go to, when in reality its a measure of how many people guess wrong as to their odds of being accepted.
PhB PhD (Score:2)
Use a real metric like real managers do: lines of code/text
The whole system is wrong (Score:2)
Why not say what it really is.... (Score:2)