Please create an account to participate in the Slashdot moderation system


Forgot your password?
Programming Software Science

The Unsung Heroes of Scientific Software ( 62

An anonymous reader sends this news from Nature: For researchers who code, academic norms for tracking the value of their work seem grossly unfair. They can spend hours contributing to software that underpins research, but if that work does not result in the authorship of a research paper and accompanying citations, there is little way to measure its impact. ... Enter Depsy, a free website launched in November 2015 that aims to "measure the value of software that powers science."

[Postdoc researcher Klaus] Schliep's profile on that site shows that he has contributed in part to seven software packages, and that he shares 34% of the credit for phangorn. Those packages have together received more than 2,600 downloads, have been cited in 89 open-access research papers and have been heavily recycled for use in other software — putting Schliep in the 99th percentile of all coders on the site by impact.

This discussion has been archived. No new comments can be posted.

The Unsung Heroes of Scientific Software

Comments Filter:
  • by muecksteiner ( 102093 ) on Wednesday January 06, 2016 @06:32AM (#51247139)

    And yes, we have to fix this somehow, sooner or later. At least in my area (Computer Graphics), complex, cutting-edge research increasingly builds on highly specialised software stacks that are being maintained by researchers the community. Whose efforts usually are not appropriately rewarded. An example is meshlab: that thing is hugely useful for lots of people - but in retrospect, the main author has all but described developing it as a mistake. As it cost him too much time that he would have needed elsewhere in his career efforts.

    A few guys are lucky, like Wenzel Jakob: he both wrote Mitsuba (the extremely useful research path tracer that everyone uses these days to build on), as well as a couple of high profile publications that set him up for an academic career. But in a lot of cases, even very good researchers only have the time and brainpower for one thing at a time: software *or* publications. We need both of them, but only reward one category. Bad move, systemically speaking.

    • Unless its freeware the guy gets paid for writing the stuff so he can't complain and if it is freeware then that was his choice to do it. He can't co-opt someone elses plaudits just because he created a tool they used.

      If you really believe that then perhaps you think the company that built the power sockets or the desk or made the PC motherboard should also get a mention? Where does it end? We all use things invented and built by others to do our jobs.

      • Re:No it isn't (Score:5, Insightful)

        by muecksteiner ( 102093 ) on Wednesday January 06, 2016 @07:15AM (#51247217)

        You have a good point with regard to "where does it end". But you seem to have mis-understood what this is about: this is not about automatically getting credits or co-authorship whenever someone uses the software you wrote. That would indeed be strange.

        Rather, the point they are making is that being the (co-)author of such a software should count as much as being the (co-)author a paper, with regard to getting tenure, or for general performance reviews within academia. Sort-of-similar metrics could be applied to this as they are applied to publications: how many people downloaded it (probably a fairly bad metric), how many papers mention using the software for experiments (probably better). The whole thing is a bit shaky, of course: if someone writes some fairly trivial piece of software that ends up being ubiquitous, they get lots of credit - while highly specialised software that took ages and lots of brainpower to develop scores low.

        But this dilemma is of course equally true for publications themselves: citation counts and such are not very good metrics, either.

        • by Viol8 ( 599362 )

          "Rather, the point they are making is that being the (co-)author of such a software should count as much as being the (co-)author a paper, with regard to getting tenure, or for general performance reviews within academia."

          Why? The software is a tool. The software authors had nothing to do with the original research undertaken by the scientists. You might as well make the programmers of MS Word that the paper was written on co-authors too. Its absurd.

          • Re:No it isn't (Score:4, Insightful)

            by Ambassador Kosh ( 18352 ) on Wednesday January 06, 2016 @07:39AM (#51247269)

            A lot of scientific software is extremely specific and designed to enable very specific types of research. There are many fields where there are 1 or 2 pieces of software exist and they where written inside the community.

            They most certainly should get credit for writing that software and enabling very specific research since in many cases without the software the research would not even exist.

            • Re:No it isn't (Score:4, Insightful)

              by muecksteiner ( 102093 ) on Wednesday January 06, 2016 @07:50AM (#51247283)

              Exactly this. Including the authors of Word would indeed be absurd. But the authors of software that is *directly pertinent* to the research in questions should get some sort of credit.

              Of course, this has fuzzy borders: to stay with the graphics examples, meshlab is probably too general to get much credit these days (it is a MS Word of mesh processing, so to speak). Mitsuba, on the other hand, probably counts for a lot of research papers where it was used - for a number of tasks, no comparable open-source software exists.

              • Right also word isn't free and it's as silly to credit MS as to credit Toyota for making the car that you drove to work . Crediting the guy that gave you a lift everyday and saved you an hour commute is reasonable and a better analogy

                • by delt0r ( 999393 )
                  Well have you ever wondered why medical journals have impact factors around 50? It is because medical people really do expect just about every piece of bullshit to get them on the author list.
            • by Viol8 ( 599362 )

              "They most certainly should get credit for writing that software and enabling very specific research since in many cases without the software the research would not even exist."

              Without many other things the research wouldn't exist either. Unless the software or its authors came up with the idea of the original research or had some hand in how it was conducted then they have no business being on the paper and just because some piece of software is a one off doesn't change this in the slightest.

              Perhaps the gu

              • This is a horrible comparison. Shakespeare could have done his plays in any theater.

                I am talking about custom written software that solves an EXTREMELY narrow niche and without it a lot of the research in the field would not even happen.


                • by gfxguy ( 98788 )

                  But who did the research? As I mentioned in another reply, when I was a research assistant, if some professor came up with some algorithm or criteria for a simulation and asked me to implement it, why should I get any research credit? I did get mentioned a couple of times in credits for having implemented the software, but anyone at my level could have done it... I got zero research credit in those cases, and didn't deserve any.

                  • Well the software I am talking about is the result of a lot of very difficult research that has taken many years to do. The people writing the software are doing the research.

                    There is no simple algorithm that a professor came up. It is a complex physical model and it is highly non-trivial to figure out which parts need to be modeled and why to get the correct physical behavior.

                    Others use the software to try and solve a specific problem that would have been impossible without a lot of very hard work in getti

                    • by gfxguy ( 98788 )
                      In that case I agree, but perhaps the software like that is then worthy of it's own research paper.
                    • It has had lots of research papers. However, when people use it the software/papers should also be cited.

                    • In the past I remember some software would get a small academic paper, enough for a conference. Then it would get cited many times by others who used it. But software is getting more complex now and may need more than a team of 2 or 3 grad students to develop and maintain it.

                • by Viol8 ( 599362 )

                  "This is a horrible comparison. Shakespeare could have done his plays in any theater."

                  And the people doing the research could have probably done it in Matlab given enough time. Your point is?

              • I agree that the people writing the software shouldn't necessarily be on the paper. However, I think they should still get credit for working on it. If you're in a community of scientists, all working on related projects that require complicated software, doesn't it make the most sense to have a few people develop it and distribute it to everyone else, rather than each lab having to make their own? Having one standardized version (or package, since not everyone will be on the same version) makes it easier t
            • by gfxguy ( 98788 )

              But writing software isn't research or necessarily insightful at all - if one person comes up with a theory or model or formula, and another person codes it, the person who coded it didn't contribute anything to the actual research, and any decent programmer would be able to implement someone else's algorithms. If the programmer came up with a way to test or do something in particular, then they certainly should get credit, but merely implementing someone else's research is NOT research in, and of, itself.

              • by tibit ( 1762298 )

                If one person comes up with a theory or model or formula that is not implemented at the moment, it's not even generally possible to tell if their contribution is of any use. To get results, you need an implementation, and the implementation is absolutely crucial. Even in terms of algorithmic improvements in "pure" computer science, if the real life implementations are constrained such that the algorithmic improvement doesn't matter, the value of the theoretical improvement is decreased or it simply becomes

            • by emil ( 695 )

              The Spice electrical circuit simulation software was developed in FORTRAN on several platforms (including VAX VMS) in the 1970s. I managed to compile it for Linux and Windows years ago, and I host the source and binaries [] on a laptop in my basement.

              This specific version is in many circuits textbooks - newer versions are not compatible with the syntax of this release. I see a fair amount of traffic for it. I should probably spend some time on a nicer HTML5 download page.

          • These program are often being written by academics which can cut into time doing research. But these programs are vital for the research. If this were physics and the academic spent a significant portion of the year helping to build and manage a new particle collider, do they get credit for it or get accused of wasting time instead of writing papers?

            This is not a binary freeware versus commercial software debate.

      • Re: (Score:3, Informative)

        by Anonymous Coward

        Yes, he chose to do it, but it's a choice that for many people means the end of their academic career. There's a huge amount of great scientific software coming out of the universities, but it tends to count very little toward a new faculty's tenure review. If you don't get tenure; you're fired. The unique issue for software is that it takes a very long time to write well and drains resources for its maintenance. For tenure, you're typically evaluated on your ability to get grants, research papers, depa

        • If someone is publishing 5-10 papers a year for 7 years, they don't have time to do any research, let alone write anything more complicated than Hello World in their favorite language.
          • If someone is publishing 5-10 papers a year, this means someone else is doing the actual work and that person is only authoring the papers. That brings up the topic of authorship. Whose names should go onto a paper, and in what order? The first author is sometimes a person who puts together the manuscript template, gets comments from everyone else, and searches EndNote's pubmed connection for manuscript titles that match the context without actually reading the cited article. (You know what I'm talking abou

      • Unless its freeware the guy gets paid for writing the stuff so he can't complain and if it is freeware then that was his choice to do it. He can't co-opt someone elses plaudits just because he created a tool they used.

        I think you have missed the point. It isn't a matter of "co-opting someone else's plaudits", more of getting an appropriate share of them. Perhaps that share should be a small one; or perhaps it should be dominant. The fact that it's freeware isn't really relevant. Do you think that the inventors of the alphabet, the number system, the periodic table of the elements, or the Web should be denied credit for their work just because they didn't "monetize" it? Believe it or not, there are still other values than

    • by Anonymous Coward

      We need both of them, but only reward one category. Bad move, systemically speaking

      Isn't this a broader problem in academia? Where I work there is huge pressure for teaching commitments, and zero recognition for it. The same goes for admin: one of the senior "makes the departmental world go round" jobs was unfilled for months because nobody wanted it, everyone knew it was a huge time-sink which took away from research and grant proposals but also came with zero recognition. Eventually the Head of Schoo

  • by Anonymous Coward

    ...of the software and then request citations as parts of the license agreement. It's not a guarantee, and yes some will fail to cite properly, but at a present a short (possibly even conference) paper with a citation request really is you're best bet to get some credit.

    The difficult part is finding a journal that will accept a description-of-software type paper *and* has a decent ranking. I'm somewhat lucky to have such a journal in my field, but I know that other fields are not so fortunate.

    • You don't have to mess with your license at all. Scientists are good at citing things if you give them something to cite.

      We try to publish a few papers yearly about new aspects of our software... and the scientists that use those pieces of the software naturally cite this papers without issue. We post our citations on our website and many people also email the mailing list to ask for the appropriate thing to cite when they're publishing findings based on our software.

  • Scientific Linux is a distribution along the lines of CentOS that is sponsored by Fermilab.

    From their about page:

    Scientific Linux is a Fermilab sponsored project. Our primary user base is within the High Energy and High Intensity Physics community. However, our users come from a wide variety of industries with various use cases all over the globe – and sometimes off of it!

    Our Mission: Driven by Fermilab’s scientific mission and focusing on the changing needs of experimental facilities, Scienti

    • by mwvdlee ( 775178 )

      Depsy currently only scans Python and R projects.
      It's probably going to take a bit more processing power to include all popular programming languages.

    • by lorinc ( 2470890 )

      SL was so much outdated that we ditched it out of our cluster 2 years ago and replaced it with ubuntu server. Having to manually install a compiler that handles C++11 in 2014 is pretty bad. Hell, we even ditched it out of the whole CS department. I grant you it's robust (even if we had some dirty problems), but it's too outdated to be usefull.

  • No need for heroes (Score:5, Insightful)

    by lorinc ( 2470890 ) on Wednesday January 06, 2016 @07:51AM (#51247285) Homepage Journal

    Please stop the bullshit of heroes with superpower in science, as science never was about glorifying people and personality cults. Leave that to the entertainment industry.

    The whole science star system is doing much more harm than good to the actual scientific outcomes.

    People tend to optimize the metric that is used. If that metric is popularity, they'll do as much as they can to become popular, with no correlation whatsoever to the importance of their original field. Publishing crappy results on a new dataset so that everyone can beat you gets you more citations than providing insights to why some methods work and some don't. Publishing a shitty software that allows a million master student to make up wrong results for the master's thesis get you more download than writing a correct implementation of uncommon algorithms.

    I went into science because I didn't give a shit about the smoke and mirrors that are so important in other fields. Most of my best technical students now are just disgusted by the "appearance prevails" mentality that is at the core of other disciplines. Please leave science as it ought to be: efficient but careless about the image.

  • by friedmud ( 512466 ) on Wednesday January 06, 2016 @12:08PM (#51248885)

    How do you get listed? My software project isn't listed. We have a few hundred users and we're nearing about 1000 citations. It's an open source project on GitHub.

    How do we tell Depsy about it?

Thufir's a Harkonnen now.