Cassini Geyser-Tasting a Bust

Maggie McKee writes "The Cassini spacecraft flew into the icy geysers erupting from Saturn's moon Enceladus on Wednesday in an attempt to figure out what they were made of, but a glitch prevented the probe from actually 'tasting' the plumes. An 'unexplained software hiccup' put the Cosmic Dust Analyzer (CDA) out of commission. Ironically, new software designed to improve the ability of the CDA to count particle hits may be to blame. Mission managers may try to re-attempt the plume fly-through later this year."

This stuff doesn't bode well for software

[Phat_Tony]

It doesn't give me much confidence that we're heading towards applications and operating systems that won't crash anytime soon when we can't even get something this important right.

It really makes me curious about the whole software quality assurance program at NASA these days. I'd like to know what their procedures are for code writing, debugging, and testing, that we're spending millions to conduct this research and apparently missing our opportunities due to software bugs.

Re:This stuff doesn't bode well for software

[iluvcapra]

This is just one data point in a rather big history. At least they didn't confuse feet-per-second with meters-per-second; at least they didn't cause their CPU to thrash due to a radar being left on and overloading the interrupts. Also, this is the same organization that managed to put two quite-autonomous rovers on Mars and keep them rolling for, what is it now?, 4 years. When one of the rovers did have a software failure, and a really bad mission-killing one, they were able to debug it and update firmware OTA from light-minutes distance, on a machine that was only intermittently alive.

They screw things up, but they seem to do very well at fault-tolerance and recovery, and I think if I were in automated systems, I'd wanna be at NASA over anywhere else, period.

Re:This stuff doesn't bode well for software

[LMacG]

Seriously? Somebody modified a program so that a system designed to do one thing could do something else and sent the modifications millions of miles across space on a radio link. There's probably not much chance of a three tier development/test/production environment here.

In the meantime, the overall Cassini project has already been incredibly successful; the happy little Mars rovers have gotten unstuck by virtue of some pretty good software hacks, but you, "Phat Tony", call into question NASA's procedures.

Seriously?

Re:This stuff doesn't bode well for software

[necro81]

These craft - their software, hardware, and the interactions between them - are so complex that there is no way to exhaustively test everything. It's complex enough that you can't even determine what an exhaustive test criteria would be. If we wanted exhaustive testing to ensure that nothing wrong ever happens, we'd never get anything off the ground. Mistakes happen, the unforeseen happens, and when communications take hours to go through, it is just plain hard. You live with it, correct mistakes as they happen, and make the best of it. They'll get a chance to try again. They have already logged tremendous amounts of data that couldn't have been gotten any other way - it's not like the whole $1.5b mission is a bust. This probe, the largest and most complex NASA has ever launched, has been operating continuously, with very few problems and no critical failures, for over a decade now.

NASA, in general, is a lot more stringent with its software than most organizations. If you would like to know more about it, you could start here [nasa.gov].

Re:This stuff doesn't bode well for software

[LiquidCoooled]

100% agree.
My sig explains the human factor quite well, what makes NASA engineers stand out above the rest is just how often they manage to carry on regardless.
In situations where normal people would give up they find a solution.

Re:This stuff doesn't bode well for software

[CheshireCatCO]

Why are you ignoring the Japanese, Chinese, Indians, Russians, and Europeans for baseline comparisons? Not to mention our own home-grown private industries like Ball, Boeing, and Lockheed-Martin. Or perhaps you'd prefer the military (as the grandfather post suggest), which also flies spacecraft?

Re:This stuff doesn't bode well for software

[CheshireCatCO]

So maybe the lesson here is "spaceflight is hard," and not "NASA sucks"? You're talking about writing software for custom-built hardware to do things that no Earth-based software has to do. And it's not like you can beta test stuff out like Mozilla does, either. If there's some obscure combination of hardware and software settings that will lead to a glitch, but everything is fine otherwise, it'll be damn hard to locate without spending many millions more for extremely extensive testing. There's a point of diminishing returns, perfection simply doesn't happen.