Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Science

Does XKCD's Cartoon Show How Scientific Publishing Is a Joke? (theatlantic.com) 133

"An XKCD comic — and its many remixes — perfectly captures the absurdity of academic research," writes the Atlantic (in an article shared by Slashdot reader shanen).

It argues that the cartoon "captured the attention of scientists — and inspired many to create versions specific to their own disciplines. Together, these became a global, interdisciplinary conversation about the nature of modern research practices." It depicts a taxonomy of the 12 "Types of Scientific Paper," presented in a grid. "The immune system is at it again," one paper's title reads. "My colleague is wrong and I can finally prove it," declares another. The gag reveals how research literature, when stripped of its jargon, is just as susceptible to repetition, triviality, pandering, and pettiness as other forms of communication. The cartoon's childlike simplicity, though, seemed to offer cover for scientists to critique and celebrate their work at the same time...

You couldn't keep the biologists away from the fun ("New microscope!! Yours is now obsolete"), and — in their usual fashion — the science journalists soon followed ("Readers love animals"). A doctoral student cobbled together a website to help users generate their own versions. We reached Peak Meme with the creation of a meta-meme outlining a taxonomy of academic-paper memes. At that point, the writer and internet activist Cory Doctorow lauded the collective project of producing these jokes as "an act of wry, insightful auto-ethnography — self-criticism wrapped in humor that tells a story."

Put another way: The joke was on target. "The meme hits the right nerve," says Vinay Prasad, an associate epidemiology professor and a prominent critic of medical research. "Many papers serve no purpose, advance no agenda, may not be correct, make no sense, and are poorly read. But they are required for promotion." The scholarly literature in many fields is riddled with extraneous work; indeed, I've always been intrigued by the idea that this sorry outcome was more or less inevitable, given the incentives at play. Take a bunch of clever, ambitious people and tell them to get as many papers published as possible while still technically passing muster through peer review ... and what do you think is going to happen? Of course the system gets gamed: The results from one experiment get sliced up into a dozen papers, statistics are massaged to produce more interesting results, and conclusions become exaggerated. The most prolific authors have found a way to publish more than one scientific paper a week. Those who can't keep up might hire a paper mill to do (or fake) the work on their behalf.

The article argues the Covid-19 pandemic induced medical journals to forego papers about large-scale clinical trials while "rapidly accepting reports that described just a handful of patients. More than a few CVs were beefed up along the way."

But pandemic publishing has only served to exacerbate some well-established bad habits, Michael Johansen, a family-medicine physician and researcher who has criticized many studies as being of minimal value, told me. "COVID publications appear to be representative of the literature at large: a few really important papers and a whole bunch of stuff that isn't or shouldn't be read."
Unfortunately, the Atlantic adds, "none of the scientists I talked with could think of a realistic solution."
This discussion has been archived. No new comments can be posted.

Does XKCD's Cartoon Show How Scientific Publishing Is a Joke?

Comments Filter:
  • You will not go to space today.
    • Not 100% of it.

      A lot of papers are written just because somebody's tenure depends on them writing X number of papers per year... but not all of them.

      • Precisely.

        "Some X are A." != "All X are A.".

        It is the same type of "logic" as "All Jews ...". (...to use a hyperbole that should end this thread. :)

  • by shanen ( 462549 ) on Saturday May 08, 2021 @08:48PM (#61364072) Homepage Journal

    Hmm... Must be a slow news day, but I confess to submitting this idea a few days ago, and I still think it's funny. Some of the derivative versions were closer to hilarious, but they lose difficulty points for being derived. However my own extension wasn't included by the editor, so I'll add it on now as a challenge for the reader:

    One of the links from The Atlantic will take you to the generator. [The link just says "website", which pretty much lacks meaning in this general context.] It can be used to somewhat easily create a version for the kinds of stories that then get linked to from Slashdot. Unfortunately, I lack the kind of imagination to do the deed, except that I think one of the stories on the last line should feature Cowboy Neal. However my idea would be feeble clickbait: "25 Things You Need to Know about Cowboy Neal" or something along those lines.

    But I've already established my lack of a sense of humor, though I think the better Slashdot discussions in days of yore started with an imaginative joke.

    • 1. Some backwater nation is using antiquated technology. 2. Apple releases some new shiny 3. Everyone, everywhere is being surveilled, hacked, monetized, advertised, extorted by corporations, their government, foreign governments and Russian hackers. 4. Some new technology was put to some novel use. 5. Some old technology is still being used 6. Elon farted 7. Bill burped 8. ...
    • I could do the same for "Management Theories" and the endless detritus that seems to flow from that. The current flavor is "Leadership". In monolithic bureaucracies. Yeah.

      Kinda miss the honesty of "do what I say or I will fire you".

      Apparently "don't be dick" hasn't made its way to the collective zeitgeist of managers. I'm sure it will have at least a thousand books published proclaiming how revolutionary it is.

      • by shanen ( 462549 )

        Well, just on the quoted parts you have the kernels for four entries. However you'll need to flesh them out for longer entries. Most of my work in creating the example was adjusting the wording so that the new titles properly filled in the available spaces. His program is basically taking the XKCD cartoon and laying new captions over it.

    • It's funny, but it does not have some grand meaning behind it. It's become fodder for the modern anti-science and anti-academic attitudes.

      • by shanen ( 462549 )

        I rather doubt that the anti-science crowd reads XKCD. The Atlantic story certainly portrayed it as scientists willing to laugh at their own pretensions. (Ditto the anti-academic crowd, though I think it's almost the same people.)

        • The thing is, the anti-science crowd is strong on Slashdot. Any scientist is automatically more stupid than a slashdotter if they have a finding that does not match what the slashdotter was taught in grammar school. An attitude that all experts are automatically wrong, based upon a reading of only the article's headline, or the dumb overly simplified paragraph written by the slashdot editor. Won't bother reading the papers before declaring them wrong. Slashdot is not about scientists for the most part, i

          • by shanen ( 462549 )

            Basically concurrence, but I think that's one of the negative changes in Slashdot over the years.

  • News at 11!

    I mean, I guess we could let AI's do all the research, but I really don't think they're going to be any better. As long as "# of publications" is a performance metric, folks are going to find ways to easily maximize their publications.

    Maybe I should write that up and submit it to a journal. I'll just "borrow" all the illustrations.

    • by cruff ( 171569 ) on Saturday May 08, 2021 @09:00PM (#61364102)

      As long as "# of publications" is a performance metric, folks are going to find ways to easily maximize their publications.

      That's like tracking the percentage of Agile sprints where the team makes their commitment as a performance metric. It leads to intentional under-commitment to preserve bonuses.

    • I mean, I guess we could let AI's do all the research

      No, because the problem is not with the research but how the presentation of it is being gamed to satisfy the publish-or-perish model. Where an AI could come in handy is inhaling all the publications in a field and reducing them to a series of useful summaries to inform researchers in that discipline. Think of it as being a super-editor.

      Such an AI should be as doable as today's plethora of natural language translators.

    • by dvice ( 6309704 )

      > I guess we could let AI's do all the research, but I really don't think they're going to be any better.

      AI has solved protein folding problem, which humans have tried to solve for years. Mind you that the AI still needs a lot of human work to do that level of research.

      > As long as "# of publications" is a performance metric,

      This is actually a good metric. Study has shown that the best publication comes on average during the time when person publishes the most of the papers. In other words, the more y

      • by Rockoon ( 1252108 ) on Sunday May 09, 2021 @01:18AM (#61364574)

        AI has solved protein folding problem, which humans have tried to solve for years.

        No, humans solved the protein folding problem, using technology as usual.

      • This is actually a good metric. Study has shown that the best publication comes on average during the time when person publishes the most of the papers. In other words, the more you publish, the more likely you are getting something right. ...

        You're assuming some sort of independence of events here that I do not think is the case. You're also measuring under the current system where people have to publish lots of papers.

    • Humans Game System to Score Higher

      Many scientists, especially the good ones hate gaming the system. They hate losing their job and entire career slightly more though.

      Many of the worse ones seem to relish the game.

    • The scientists I know all do it as a labour of love. Social status among peers in THE most important thing. Getting paid is a bonus. They play the game because they have to. For example, in the USA & Canada, around 75% of faculty don't have tenure & so are living from short-term contract to short-term contract. They'd love to publish fewer papers of higher quality but they're afraid of not getting the next short-term contract. At the moment, there's an exodus of researchers from academia (some of wh

  • by joe_frisch ( 1366229 ) on Saturday May 08, 2021 @08:58PM (#61364096)
    The root problem is that funding agencies, which are generally run by non-experts, want a way to judge the work of scientists and laboratories. They have settled on publication statistics as a way to do that. Its a bad measure, but its not clear what better measure is available. It is rare to find a really talented scientist who has risen high enough in the bureaucracy to control funding - most scientists can't stomach the effort to get there. So while there are a lot of very good papers, there are also publications just done to game the system. Wish there was a better system available.
    • Simple solution. Unlimited funding.

      • Seriously considering the market communism (essentially the Bell Labs model) for research.

        It's become glaringly obvious the limitations of market based research, and even public-private partnerships seem little more than another way to drain the taxpayers dry.

        Which would even be tolerable if you weren't paying through the nose for the product at the end.(hey pharmaceutical companies).

        I could foresee government going back into basic research.

      • by dvice ( 6309704 )

        Considering what is best for humanity, statistically best research funding strategy is to divide it into as many people as possible. Downside of that is that large research projects become impossible.

        Fusion research is a good example. It has not been possible without large funding. But even with small funding, a lot of alternative energy technologies have been developed. I would prefer that have both large funding for this kind of super projects and small funding to many projects.

        If I had the money, I would

    • by ceoyoyo ( 59147 ) on Saturday May 08, 2021 @11:50PM (#61364412)

      Funding agencies make most, if not all, of their funding decisions via peer review. Any decisions that aren't made that way are made over the usual golf and blow jobs. They don't count papers.

      Universities want to evaluate their staff. It's they who've fallen into the widespread trap that metrics == good. So now they've got metrics.

      The scientists the Atlantic talked to who can't think of a solution aren't thinking very hard. Just stop using paper count to mean anything. Yeah, you'll have to work a little harder to evaluate your staff's performance. You might have to get external reviewers in. It's hard to do that every quarter. That's a bonus. Do it every five years instead. Then you encourage actual productive work on projects with a reasonable timeframe.

      • by Rockoon ( 1252108 ) on Sunday May 09, 2021 @01:34AM (#61364604)
        Give people credit for replication. That is, not doing replication, but having their work be replicated.

        Andy published a paper in a new area of study, showing how A leads to B.
        Beth publishes a paper, replicating Andy's work, confirming that A leads to B, and further shows that A can also lead to C.

        It is at the point of Beth's paper being published that Andy should get any credit.

        Now, some might say that this is problematic because not all science can be replicated by any old tom, dick, and harry. Some science is expensive.
        My response is so fucking what. The people doing expensive science no longer need credit at every step, and its a team of people at that point.

        Also, sometimes nobody cares to replicate because what was discovered is useless if true. My response is again so fucking what. Thems the breaks. How much credit should you get for discovering the quite unremarkable melting point of an amalgam that never existed until you made some? Hey maybe someone will replicate one day. Until then fuck off with taking credit.
        • by VeryFluffyBunny ( 5037285 ) on Sunday May 09, 2021 @05:42AM (#61364916)

          It is at the point of Beth's paper being published that Andy should get any credit.

          That's how it already works - researchers' impact factor is calculated from how often their papers are cited & quality of the journals they're published in.

          There's 2 problems with this: #1 Researchers can still game the system by publishing claims that are guaranteed to incite indignant outrage among their peers thereby guaranteeing a flurry or rebuttal papers which in turn generates a large number of citations. #2 Rating papers by who publishes them gives even more power to the parasitic academic publishing industry, e.g. Elsevier, which is currently bleeding academia dry & making it more difficult for researchers to do their work.

          All solutions that simply exchange one set of metrics for another is missing the fundamental point that metrics alone are a very poor solution, as was expressed by Charles Goodhart (1975):

          "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

          Otherwise known as: When a measure becomes a target, it ceases to be a good measure.

          Reference: Goodhart, Charles. ‘Problems of Monetary Management: The U.K. Experience’. In Inflation, Depression, and Economic Policy in the West, edited by Anthony S Courakis, 111–46. Rowman & Littlefield, 1981. https://link.springer.com/chap... [springer.com].

          • That's how it already works - researchers' impact factor is calculated from how often their papers are cited & quality of the journals they're published in.

            You say thats how it works, then describe how it actually works. Right now they get credit for citations. Citations are not replication. Citations can in fact be the exact opposite of replication, but more often than not a citation means that this new study refines a value from that older study, which is frequently not done by any replication of any kind but instead comes from a different angle entirely.

            You show that A leads to B.
            Someone else shows that Z leads to A.
            br You dont get any credit. You get

    • No, it's not about stomaching effort. It's just in this plutocratic system scientists aren't deemed worthy of determining where funds go on their own. Of course they might make wrong decisions, but plutocrats that aren't into science are even more likely to, just like metrics devised by said plutocrats would naturally lead to outcomes unrelated to encouraging good science. If even giving rewards for rat tails in order to purge them actually leads to rat farming then there is nothing surprising that similar
      • There is a lot of truth to that. At the very least I think the decisions should be made at a lower level. Still, there is finite funding and somehow the money needs to be divided between people / projects.
    • Well a partial solution could be if the people with the money would include "replication of studies that have as of yet not been replicated" as equal weight as a new publication. Actually that would also solve the replication crisis.
  • by account_deleted ( 4530225 ) on Saturday May 08, 2021 @09:21PM (#61364148)
    Comment removed based on user account deletion
    • by shanen ( 462549 )

      So far I'm apparently the only person who generated a new one?

      • by DeBaas ( 470886 )

        Maybe, but I just created one [twitter.com] as well on programming languages

        • by shanen ( 462549 )

          See? That's how to write comedy. The #3, #4, #7, and #11 were especially funny. But I noticed two typos, too.

          Alas, I never have a givable mod point. Maybe it's related to my "Contradictor" achievement? (I noticed this submission was my 2^5 achievement in that category.)

  • Too close to the bone.

  • by quantaman ( 517394 ) on Saturday May 08, 2021 @10:17PM (#61364218)

    It's a good comic but you can do the same thing with TV shows, movies, new products, etc, etc. Things, in general, are prone to patterns and categorization.

    The fact he needed 12 categories actually demonstrates an impressive level of diversity.

  • and work is often mundane. Regular folk tend to forget about the mundanity in science because the media focuses on big, splashy headlines.
    • Academic science rewards splashiness. Therefore there is also an abundance of scientists spinning their mundane work as ground-breaking, life-altering, and indispensable.

      Good work obviously gets done, but most information is noise, not signal, regardless of whose mouth it comes from.

      The "science cheerleader" types (mostly in media, but also academic Twitter) who assert that any science is all those things simply by virtue of being science, don't help good thinking, or good science, which frequently requires

      • by sfcat ( 872532 )

        Academic science rewards splashiness. Therefore there is also an abundance of scientists spinning their mundane work as ground-breaking, life-altering, and indispensable.

        No, that's lazy administration that rewards that. Science rewards reproducibility and usefulness. The problem is that science is one of the biggest victims of name-dropping right now (besides various religious figures). A lot gets said and done in the name of science when there is no scientific reason or research to believe some of those things (energy and environmental policy are examples). Also, science and statistics are hard and some scientists (and even some entire fields) really don't do the relev

      • Every one rewards splashiness. It gets more attention.

  • by bradley13 ( 1118935 ) on Sunday May 09, 2021 @12:34AM (#61364510) Homepage
    Look, a good researcher might have something really worth publishing once a year. In some fields, maybe only every 2-3 years. A PhD student ought to get a paper out of their thesis. Maybe two, if they really do something substantial. The current academic world expects continuous publications. It gets worse: I'm a professor at a teaching college. Our accreditation requires us to publish, which is just brain-dead stupid for most of the faculty. As far as I can tell, this stems from classic bad management. Administrators want a simple number they can judge people by, so they count publications.
  • by drkshadow ( 6277460 ) on Sunday May 09, 2021 @01:01AM (#61364546)

    # "One, Two, ???, Profit!"
    # "Well duh, this is clearly obvious"
    # "This might be obvious for you, but some other people..."
    # "Repost!"
    # "You're thinking about it from the wrong view point"
    # "<Technical details>"
    # "If we can do this, then imagine if we..."
    # First post - everyone tries to be on top
    ## 75 Replies

  • Obscurantists will take this satire at face value and will see it as a proof that their Facebook group was right all along.
  • Sad to see how xkcd is still not touch friendly :(
    • I agree with that. Fortunately my RSS reader of choice adds a "hover" button thtat displays the alt text that goes with the image.
  • I prefer "The Endochronic Properties of Resublimated Thiotimoline".

  • PROTIP: Randall Munroe isn't particularly smart or an expert in most of what he crudely draws about. He's just a one-eyed among the blind.

    Yes, your headline and its ridiculous generalization for the sake of an uneducated ideology-motivated anti-science agenda backs up me calling you blind very well.

    PROTIP2: Don't take it seriously, it's a cartoon! You're cargo culting science, one step away from being a flat-earther. So you should not be talking.

  • As a former scientist working on quantum computing:

    Half of the type of papers was involved in the evolution of superconducting QC, and now they work. They probably would not work I there had not been a lot of people going down every road, looking at old papers, looking at materials which were unknown ho they would behave, attempts to transfer theoretical ideas.

    And yes the "the task I had to do anyway was so hard that I wrote a paper about it" is the type of paper which made QC possible.

  • by WierdUncle ( 6807634 ) on Sunday May 09, 2021 @09:43AM (#61365552)

    Long ago, I did a final year project in electronics, which involved background research into relevant papers on switched-capacitor filters. The amount of dross was amazing, even then. Eventually, I found that the derivative dross papers all cited one or two papers of actual interest. A typical dross paper would pick up on some formulae in the original work, twiddle the algebra a bit, and derive some load of rubbish results that had little connection to reality. It was not wrong, as such, but just fundamentally useless.

    A few years later, the same phenomenon came up, when my father did a bit of consultancy after he had retired as a scientific civil servant. He had been given some papers by his customer (our neighbour), and wanted my opinion. There was one paper worth studying. The rest were just derivative and not at all informative. He was relieved when I told him that. He thought he was going senile, because that sort of rubbish would never have been published in his day.

    • by john83 ( 923470 )
      I realised at some point in my 20s that I had published as many papers as Richard Feynman did in his entire career. While I'm not an expert on his work, I'm going to go out on a limb here and suggest that his contribution to science is greater than mine. Paper counts are crap. Citation counts are crap. The may not have been complete crap once, but they're performance metrics now and are gamed to death by very clever people. Someone will eventually move us on to new metrics. Those will quickly become crap to
  • Obligatory XKCD reference: https://xkcd.com/2456/ [xkcd.com]
  • Take Prof. Sun's work. He got his university to post on their website ( https://news.northeastern.edu/... [northeastern.edu] ) and then he touted it on linkedin. When asked for details he said 'no, waiting for IP'. So, here it is 10 months later and still no product. But i will bet real money he got more funding. The way science is funded is broken. It has been for awhile.

  • is doing the same thing to patents as the publishing requirement did to papers. The NSF started looking for inventions being actually used in the real world so they asked for technology transfer plans in grant proposals. Sometimes that resulted in open sourcing (often useless) code or collaborations with industry (who often collaborate for the publicity/PR instead of the research results). Sometimes it results in patents filed just to check the technology transfer box on grant reports. Does not even ma
  • Because
    1. Back 50 years ago, 70% of instructors in universities and colleges were tenure, or
    tenure track. Now it's under 30%, and everyone else has no guarantee they'll
    have a job in the fall.
    2. Colleges and universities DO NOT CARE, in general, about the quality of the

We are each entitled to our own opinion, but no one is entitled to his own facts. -- Patrick Moynihan

Working...