Computer Program For Particle Physics At Risk of Obsolescence (quantamagazine.org) 105
"Maintenance of the software that's used for the hardest physics calculations rests almost entirely with a retiree," reports Quanta magazine, saying the situation "reveals the problematic incentive structure of academia."
Particle physicists use some of the longest equations in all of science. To look for signs of new elementary particles in collisions at the Large Hadron Collider, for example, they draw thousands of pictures called Feynman diagrams that depict possible collision outcomes, each one encoding a complicated formula that can be millions of terms long. Summing formulas like these with pen and paper is impossible; even adding them with computers is a challenge. The algebra rules we learn in school are fast enough for homework, but for particle physics they are woefully inefficient.
Programs called computer algebra systems strive to handle these tasks. And if you want to solve the biggest equations in the world, for 33 years one program has stood out: FORM. Developed by the Dutch particle physicist Jos Vermaseren, FORM is a key part of the infrastructure of particle physics, necessary for the hardest calculations. However, as with surprisingly many essential pieces of digital infrastructure, FORM's maintenance rests largely on one person: Vermaseren himself. And at 73, Vermaseren has begun to step back from FORM development. Due to the incentive structure of academia, which prizes published papers, not software tools, no successor has emerged. If the situation does not change, particle physics may be forced to slow down dramatically...
Without ongoing development, FORM will get less and less usable — only able to interact with older computer code, and not aligned with how today's students learn to program. Experienced users will stick with it, but younger researchers will adopt alternative computer algebra programs like Mathematica that are more user-friendly but orders of magnitude slower. In practice, many of these physicists will decide that certain problems are off-limits — too difficult to handle. So particle physics will stall, with only a few people able to work on the hardest calculations.
In April, Vermaseren is holding a summit of FORM users to plan for the future. They will discuss how to keep FORM alive: how to maintain and extend it, and how to show a new generation of students just how much it can do. With luck, hard work and funding, they may preserve one of the most powerful tools in physics.
Thanks to long-time Slashdot reader g01d4 for submitting the story.
Programs called computer algebra systems strive to handle these tasks. And if you want to solve the biggest equations in the world, for 33 years one program has stood out: FORM. Developed by the Dutch particle physicist Jos Vermaseren, FORM is a key part of the infrastructure of particle physics, necessary for the hardest calculations. However, as with surprisingly many essential pieces of digital infrastructure, FORM's maintenance rests largely on one person: Vermaseren himself. And at 73, Vermaseren has begun to step back from FORM development. Due to the incentive structure of academia, which prizes published papers, not software tools, no successor has emerged. If the situation does not change, particle physics may be forced to slow down dramatically...
Without ongoing development, FORM will get less and less usable — only able to interact with older computer code, and not aligned with how today's students learn to program. Experienced users will stick with it, but younger researchers will adopt alternative computer algebra programs like Mathematica that are more user-friendly but orders of magnitude slower. In practice, many of these physicists will decide that certain problems are off-limits — too difficult to handle. So particle physics will stall, with only a few people able to work on the hardest calculations.
In April, Vermaseren is holding a summit of FORM users to plan for the future. They will discuss how to keep FORM alive: how to maintain and extend it, and how to show a new generation of students just how much it can do. With luck, hard work and funding, they may preserve one of the most powerful tools in physics.
Thanks to long-time Slashdot reader g01d4 for submitting the story.
Obvious question (Score:2)
Is it open source?
If so, you don't need particle physicists to maintain it. You only need particle physicists to articulate their specific needs. Software engineers and high-flying mathematicians will be better at modernising the UI and squeezing out a few extra cycles at each step of the calculation.
Re:Obvious question (Score:4, Informative)
Re: Obvious question (Score:1)
Re: Obvious question (Score:5, Insightful)
Maybe, just maybe, you are not so good at explaining things to others.
Assume for the moment that the domain experts are seated just across the table from the software people. In something like an (shudder) Agile Scrum or whatever. It is as much the responsibility of the software experts to question the domain people when there is a hole in the specification or something that they don't understand.
When I, as a domain expert, find myself in meetings with s/w people who just push a requirements specification template across the table and expect me to fill it out, they are often MBAs sitting across from me. Who probably don't know squat about actually building an app and couldn't spot a missing piece of information that the coders will need. In other words, they expect me to be a coder*. Their job consists of obtaining the spec, having it translated into Hindi and forwarding it to the development subcontractor.
*In many cases, it takes longer to write a contractually sound** spec than it does to just write the damned app.
**By contractually sound, I mean something that can be handed to a subcontractor without incurring too many change orders, associated additional fees and legal department involvement. As opposed to something done in house, where the coder can just wander into my cubicle and ask me to "Please explain this shit!" with no penalties or delays.
Re: (Score:2, Insightful)
Re: (Score:1)
If there is something you forgot about, and that almost always happens, then that is on you for any delays that causes.
I don't know what you need to do your job. What do you think I am? A mind reader?
Re: (Score:3)
Re: Obvious question (Score:2)
Re: Obvious question (Score:4, Insightful)
Interesting discussion (including SFcat's genuine frustration as someone taking the other side in the debate).
At my old job I switched groups, going from kind of a general coder to support production code to a data analyst specializing in SQL who helped test production code my new group specified to the developers. So I think I've dealt with BS on both sides of this divide...
As a coder, it was incredibly frustrating to get half-baked specs from business people who thought all our data was in a magic excel spreadsheet and that programming was basically an advanced level of excel monkeying. They would also try to move the goalposts at various stages. The secret, unsurprisingly, was good communication. I really dumbed down the explanation of what pseudocoding is, and most of the business types got it. We have to translate from english to computer code, so we need this intermediate document that's kind of a mix, and we need to have an organized way where if one side wants to change something on that agreed-upon spec, the other side can push back. The key attitude to have/project is _working together._ But did I also want to kill certain business people practically every week? Hell yes!
As an analyst, I definitely saw all the things you and PPH mentioned. I was a little bit shocked how the attitude of the fulltime Java devs was to rush out 'working' prototypes that had absolutely no re-usability for the finished product. There was also just frequent and bordering-on-absurd breakdowns in communication. You are 100% right about modern coders being fixated on code that compiles and spits out an answer. In fact, I guess I had a job more or less doing the sanity checks they should have done! And a lot of being able to do sanity checks is actually understanding the greater context of the problem... which the devs clearly didn't do.
I don't know if they teach this in coding bootcamps or whatever the fuck, but one good idea is that as a programmer whenever you get into a situation where you have to make an assumption... note down the options you considered, and the reason you decided what you did in the comments, along with the date. Immediately call the person who provided the spec and ask. Note down their response in the same comment, and write in their name and the date/time you called. This both covers your ass and raises your value to the person writing the spec... who will quickly realize they want to work with YOU versus someone else.
Re: (Score:2)
In my organization, we have a similar... table arrangement.
Though I wouldn't refer to me and my CTO as "s/w people."
We're software engineers with 4 year computer science degrees, and broad elective studies under our belt on top of that.
I'll grant you that I don't really know what a CS degree is worth these days, in terms of what you're actually taught to achieve it, but it in my day (circa late 90s), it required a stupid amount of electrical engineering and advanced math knowlege.
The kind of stuff
Re: (Score:2)
2) your s/w guys are engineers, and fuck you for calling them "s/w guys"
If they are engineers, where's their P.E. license? And I didn't exactly call them "s/w guys". "S/w people" as a catch-all for CS graduates, code monkeys or whatever. I hope you read specs more carefully.
You probably missed the brown M&Ms clause as well.
Re: (Score:2, Informative)
If they are engineers, where's their P.E. license?
How stupid.
An engineering graduate is still an engineer, whether or not they have acquired their PE yet, or not.
Engineers and Professional Engineers are related, but not synonymous terms.
If you are confused, here [nspe.org] is a reference you can learn so that you don't say stupid shit again in the future.
And I didn't exactly call them "s/w guys".
I actually referred to them as both.
Initially, using your terminology, and secondarily using mine with whatever built in sexist bias that life has programmed into me over the years.
I hope you are able to keep
Re: (Score:2)
Honestly the 'table' sounds like a situation where MBAs are trying to get a subject-matter expert to write pseudocode while the actual programmers work for a subcontractor in India and potentially haven't even been told they're working on the project yet.
So nobody at this table is an engineer, and it is a big stretch to call this type of MBA a 'software guy.'
Re: Obvious question (Score:2)
What you were trying to do could have been expressed through equations I assume.
I'm sure it could, but not necessarily in a mathematic system you understand well enough to implement.
Re: (Score:2)
I'm sure it could, but not necessarily in a mathematic system you understand well enough to implement.
That comment embodies why while there are plenty of programmers who could help you, none of us will. Also, you should have said mathematical instead of mathematic as you used the indefinite article.
Re: (Score:2)
Sfcat, your posts come across as trying to find fault and assign blame rather than trying to understand and provide help.
I think Zectron was talking about the wide gap between the math written in a paper and the software written in a numeric application. It can be very hard to assemble and manage a widely cross-domain team. Physicists may only know the math. HPC developers know both languages and what tools to use. GUI developers often don't know either.
I think PPH was talking about requirements for sub
Re: (Score:2)
There is a reason that there are still FORTRAN libraries in widespread use. Getting numerical calculations correct is far harder than it seems. It isn't just naively translating the algebraic expressions into similar-looking code. I am willing to bet that most programmers do not know how their double is encoded in memory, what limitations this imposes, etc. I was recently discussing integer representation - e.g. two's complement - with a programmer in his 20s... he's bright, but basically has never needed t
Re: Obvious question (Score:4, Interesting)
I am sure that particle physicists are fine at explaining what they want. Most of them also teach. The problem is that most software engineers do not want to take the time that it takes to do, say a masters and then PhD to gain the level of subject knowledge that they are likely to need.
Service level agreements and outsourcing work perfectly well for some areas, but it does depend on the complexity of the area and how quickly that area changes. It works fairly poorly in most scientific, research areas because these areas are both complex and changeable. What is needed is a deep partnership, with people working together over the years, rather than a short term contractual relationship. The computer scientists learn the scientific domain and those in the scientific domain learn the computing; some people will do so much of other discipline that they will even jump over the fence; that was my route, having a PhD level education in genetics and molecular biology, then being self-taught in computing to the point that this is now my job; I now need biologists to explain the detail to me, but my educational background means that I can ask the right questions and I can understand what they answers are.
This is hard to achieve and hard to maintain; part of this is, indeed, the incentive and career structure at Universities and it is something that we are trying to change. Importing some kind of agile methodology from business does not necessarily work. Science is not e-commerce, or business computing.
Re: (Score:2)
Try explaining quantum mechanics to a shoe. Did you succeed? No? Then you are not very good at explaining things.
Re: (Score:2)
They try to do this where I work, and it fails miserably. Trying to explain what to do to a programmer who is not a subject matter expert is like trying to explain ray tracing to a chimpanzee.
Maybe you just need more chimps and more time [wikipedia.org] ...
Re:Obvious question (Score:5, Insightful)
Re:Obvious question (Score:5, Insightful)
Wish I had mod points. I occasionally work with nuclear physicists and it would take me anything from years to a lifetime to learn what they're doing in their code and why, and how to change it without breaking things.
And to the person who keeps bleating "you just need to explain it, if you can't do that you're an idiot": We have about 6.023e23 books on writing secure software, it's been explained endlessly in hundreds of different ways, and yet we don't have anything like secure software. On top of that, security is simple compared to some of the physics we're dealing with.
Re: (Score:2, Funny)
We have about 6.023e23 books on writing secure software
All those books were written by Avogadro, who was a Mole.
Re: Obvious question (Score:2)
Re: (Score:2)
It is done in some sense, but people come up with better ways to do things. A link from the article: https://www.quantamagazine.org/the-algorithm-that-lets-particle-physicists-count-higher-than-two-20211122/
If you find a better algorithm to simplify an equation, do you write a new program or integrate into an existing one?
I don’t really believe the premise that nobody will maintain this program. It seems too widely used.
Migrating/refactoring high performance code can be tricky. You can easily do thin
Re: Obvious question (Score:2)
Re: (Score:2)
The programmer does need to have SOME subject matter expertise. And it helps for them to have actual interest in the subject.
Honestly "programmer" feels close to encompassed by "human to computer translation expert". If the subject matter expert is bad at explaining things then the programmer can't do anything to help.
And honestly I think most people are bad at explaining things. It's one reason I keep thinking we need a change in how we "teach" and "learn". Why not reuse the best methods everywhere? W
Re: Obvious question (Score:3)
rying to explain what to do to a programmer who is not a subject matter expert is like trying to explain ray tracing to a chimpanzee.
In fairness, trying to explain programming to most of the chimpanzees who call themselves "programmers" is even more difficult than explaining Astro Navigation to Arnold Rimmer.
At least Rimmer could spell "fish"!
Re: (Score:2)
Please do not disrespect chimpanzees this way.
Re: (Score:2)
By this measure, no programmer is useful (or at least efficient) in other than a field off endeavor they studied and more or less mastered.
In the business world you use analysts, Business Analysts most often titled, to understand customer needs and present these to programmers in a manner they can understand and deliver on. If this has already been explained, it ought to be again and again, lest anyone think you need to be a SME to program.
How the hell do new ideas become useable? You're a SME on the next i
Re: (Score:2)
0.So THAT;S why we learn new things...
1.I assume you have proven examples. Have them use those as reference, and work until they can replicate success...
Re:Obvious question (Score:4, Insightful)
It's highly technical.
Academia doesn't value software development like it should.
Re: (Score:2)
Academia doesn't value software development like it should.
I think it s not as bad as it seems. People writing complicated mathematical software publish papers explaining their new methods, basically they publish the release notes. This Release Notes paper gets plenty of citations when the software is used by other academics.
Debugging and GUI development is certainly under-valued for the time they take, but breakthrough mathematics/physics made into code makes good publication records for these academics. Arguably GUI development is more a side engineering than the
Re: Obvious question (Score:2)
Re: (Score:2)
Publish where, what impact factor will that journal have?
The IF of the journal does not matter, because the Release Notes is the obligatory paper you have to cite. It will have hundred or thousands of citations independently of where it was published.
* Gwyddion, GUI image processing software. http://gwyddion.net/publicatio... [gwyddion.net] Their original publication "Gwyddion: an open-source software for SPM data analysis" (2012) https://www.degruyter.com/docu... [degruyter.com] has 2663 citations right now.
* GROMACS, molecular dynamics software. The group published 200 papers: https://manual [gromacs.org]
Re: (Score:2)
Academia doesn't value software development like it should.
Academia doesn't devalue it. The funding agencies, i.e. the government, doesn't value it. If they could get money to work on it, academia would value it just fine. But the government answers to the people who voted for them who demand "value for money" so why are we giving money to slackers who aren't doing important research?
Re: (Score:3)
you don't need particle physicists to maintain it. You only need particle physicists to articulate their specific needs.
I do not think so. Bug fixes could be impossible without solid background in mathematics.
FORM (symbolic manipulation system) [wikipedia.org]
Re: (Score:1)
Re:Obvious question (Score:4, Insightful)
It seems less than advisable to use some custom piece of barely maintained code from the 80s to do your analysis work in the 2020s.
There is the argument It Works. There are millions of lines of code of legacy Fortran code that run the daily weather forecasts or nuclear bomb simulation. Nobody wants to spend millions in porting and validation, to replace something that works with something that maybe does not work anymore. The appearance of new fashionable languages is not enough compelling, when everybody in the field knows Fortran (or Cobol).
Re:Obvious question (Score:5, Informative)
Re: (Score:2)
And nobody uses the Fortran linear algebra library anymore
This is flat out not true.
because the GPU is 1000s of times faster for matrix operations than using the Fortran libraries on the CPU.
Also flat out not true. Compare a 32 core threadripper's double precision GFLOPS to a 3090's GFLOPS. The difference is big, but not thousands of times big. And that works if you can do everything on the GPU. If you need to shuttle back and forth from CPU to GPU, you lose a lot in latency.
Sometimes it's a lot faster to d
Re: (Score:3)
At least according to the article, it has been measured and it is orders of magnitude faster than Mathematica. So, at least for some problems this code seems much better.
A GPU may not be a great fit for symbolic manipulations, which is the primary purpose of this program. Perhaps any numerical calculations could be offloaded to a GPU, but it is unlikely to help in the principal use case of solving huge equations.
Re: (Score:2)
Re: (Score:2)
While one use case may be to export to Mathematica, it doesn’t encompass everything the program can do. According to the article, it can solve (I presume some subset of interesting) equations orders of magnitude faster than Mathematica. It can treat a set of equations that cannot fit into RAM. In other words, it is designed for different use cases than Mathematica. Just exporting to Mathematica will lose those particular capabilities.
Re: (Score:2)
One more thing: Mathematica is not free. This program is.
Re: (Score:1)
Re: Obvious question (Score:3)
Re: (Score:2)
It's fairly well commented, but it's extremely idiosyncratic and structured weirdly. At first glance it doesn't look to bad, but once you start going deeper it get weird/
Many single letter variable names, odd casing conventions, forward and backward gotos, etc. I mean, you can really tell it's sciency code, sucks but gets the job done.
Re: (Score:2)
Meh, still better than the NEC code. The comments are extremely helpful, if a bit sparse, which makes me hate it less.
I don't know that I'd want to maintain it as it is, but I wouldn't mind working on a clean up.
Re: (Score:2)
Lol, yeah, I was about to mention NEC. It completely blew my mind last year when I briefly got into using NEC for ham radio antenna design and discovered that pretty much every free/open-source program based on NEC is literally just a wrapper around a nearly-incomprehensible 30+ year old Fortran program that pretends it's reading and writing punched cards.
Re: (Score:3)
There are a couple of functions in there that are pretty demonic, but they're small and well documented. Someone could re-implement them to look nicer, if they really needed.
Otherwise, single-letter variable names seem to be limited to iterators, which is a good place for them.
goto use appeared reasonable from what I scanned- minus a couple of examples of pretty egregious spaghetti (what I suspect you saw)
It's not beautiful code, but it's not remotely close to horrible.
Re: (Score:2)
Obvious response: yes. Just do a quick search.
But I think your quick response is unlikely to be true. For one, the program doesn't have much of a UI (text input, text output), so that's not where the maintenance lies.
I don't know the details of the program (having just heard of it). It seems like it's mostly a symbolic manipulation program, but geared towards very large equations - equations that cannot fit into main memory of even modern systems. According to the article, it is orders of magnitude faster t
Re: (Score:2)
Re: (Score:2)
A lot of physics code is available free online, so anyone who i
Good (Score:1)
Re:Good (Score:4, Funny)
When do you start coding?
SRIM/TRIM for semiconductors (Score:1)
Re: SRIM/TRIM for semiconductors (Score:1)
Re: (Score:2)
FLOSS avoid this (Score:1)
Re:FLOSS avoid this (Score:5, Informative)
Just open source too
It is FLOSS, licensed under GPL [github.com]. FLOSS provides the opportunity to maintain the software, but as the fine article observes, it doesn't provide incentives for people to do the work. The incentives, are the problem, not the availability of the code.
Re: (Score:2)
Perhaps when the current author/maintainer dies, someone else will be more motivated to take over. Worst case it has to be run on some older systems for a little while.
Re: (Score:2)
Re: FLOSS avoid this (Score:2)
Re: (Score:2)
Papers are archaic (Score:1)
For many fields writing papers is just archaic.
A lot of value can actually be in datasets, in software or in other things.
In machine learning for example, a Jupyter notebook is often far better than some poorly written paper that's almost impossible to reproduce. (As some major papers in the past have been)
Re: Papers are archaic (Score:1)
Yeah...but even the best digital artifact is fundamentally ephemeral unless it is well documented in clear language.
That is what the platonic ideal of the paper is: explain what you did in enough detail for someone to understand, ask intelligent questions, or even reproduce something like your result.
Dumping a pile of code, even with some comments, is not valid substitute. Because there might be something you forgot to write down, or inadvertently obfuscated for reasons of efficiency or prettiness.
And peopl
Obligatory XKCD (Score:5, Insightful)
do what you normal do (Score:2)
Where is it? (Score:2)
It's not linked to in the article, where is it?
Some time ago there was an article about needing Computer Scientists to optimize the algorithms that run on HCP clusters, as they'd only been written and maintained by physics grad students. Software to simulate the whole universe, and all the various included laws.
But where is it? If one wanted to start, where would they look?
CERN (Score:2)
CERN has software developers - some famous.
CERN will also have electricity due to France's reactors.
Maybe they can talk.
Re: (Score:2)
Amplituhedron moved on from locality years ago (Score:2)
Re: (Score:1)
So this program is 33 years old? (Score:2)
Why does it need maintenance? Bitrot lol?
How can we include software work in academia? (Score:2)
Is our problem that we believe the world can and must be split into separate little boxes? And that we can only focus on one box at a time to get true expertise?
That biologists looking at genes think they can work without knowing about the software that allows their results.
The physicists looking at particle accelerator results that want to ignore all the methods they use to extract answers with.
They want to focus on the genes/particles, but aren't willing to spend the effort to maintain good tools and equ
Clearly, the answer is Rust (Score:2)
Re: (Score:2)
Put a little Rust on it. Super-safe coding. Never be a problem again.
+1 Funny
Not just particle physics. It's eveywhere (Score:1)
Anyone who's ever had to use MODTRAN, please raise your hand and suppress your tears.
There's a slick gui and everything for it, but under the hood is a mess of fortran that compiles into a giant all encompassing blob.
One controls the behavior of this blob by supplying files called "tapes" to the working directory in which the blob executes. "tape5" contains the "control cards" that determine the execution path inside the blob and "tape6" contains the program output...helpfully formatted in 72 columns with p
\o/ (Score:1)
Such a change that academia is a naturally-occurring aspect of the universe which cannot be influenced along a path which could benefit mankind.
Re: \o/ (Score:1)
Such a shame
Correcting an auto-incorrection.
GitHub link, GPL licensed (Score:3)
Link (repo): https://github.com/vermaseren/... [github.com]
Homepage: https://www.nikhef.nl/~form/ [nikhef.nl]
Written in C, using libgmp and libz. Has complete autoconf/autotools setup.
Throw more hardware at it? (Score:2)
Re: (Score:2)
Re: (Score:2)
Hmm (Score:2)
...only able to interact with older computer code,
So, rewrite the older computer code, if it's even worth doing.
and not aligned with how today's students learn to program
This reminds me of when my little nephew asked why all the clocks in the world aren't replaced with easier-to-read digital clocks. Shouldn't the new students be learning the better tools if they need to use them in their field?
Re: (Score:2)
Luminiferous ether? What poppycock. Why it is the tenebriferous ether that suffices all space and time.
Re: (Score:2)
tenebriferous -- well played. I had to look it up, and not many references on the intertubes.
Re: (Score:2)
Re: Particle Physics should end anyway. (Score:1)
Re: (Score:2)
Flash radiotherapy not enough for you? There are others. How about 3d X-ray scanning?