Aglassis writes "NASA investigators have determined that a software update performed in June of 2006 may have doomed the 10-year-old spacecraft. Apparently the software error caused the solar arrays to drive against a mechanical stop which then forced the spacecraft into safe mode. Unfortunately, after that the spacecraft's radiator was pointed at the sun which overheated the battery and destroyed it. Contact was lost with the Mars Global Surveyor spacecraft in November 2006. NASA will form an internal review board to determine formally the cause of the loss of the spacecraft and what remedial actions are needed for future missions."
I don't believe it. Its most likely the Martian automated defense system setup just before we sent a probe and destroyed their civilisation [slashdot.org].
What you need to do is hold back on producing all those "fun" bugs that we all introduce into systems until you've the reputation as one of the best coders in the world, then go work for NASA and just go wild on some system that won't be used until it's in deep space and you're off working for Google, having destroyed the paper trail.
...But if they installed the update on a gentoo sandbox before installing it on the MGS itself, it wouldn't be compiled for EXACTLY that machine, and as we all know, it's the precise compiling that results in gentoo's 20% performance increase (that and funrolling loops and putting flashy stripes on the computer, along with maybe a 8" exhaust).
I don't know. And people with their "keyboard" and "mouse." Idiots I say. The only true way to interact with a computer is by plugging wires into the serial port and generating the necessary electrical pulses myself.
We used to live in a vacuum tube. When the computer was running, and your bit was accessed, you almost had enough light to read by. Mother would disconnect the tube when she went to bed, causing floating point errors for almost eight clock-cycles...
Why don't computers use NASA-quality hardware, ready for space? Why don't all computers use just a single configuration (peripherals, cards, interfaces)?
The purpose of an operating system is so much wider than what the Mars Global Surveyor had to do.
Actually, they buy their OS's off the shelf. (VxWorks for the rovers, for example)
That said, you could get software written to this level of perfection if you wanted. It's easy- follow the space shuttle's team's example. You have a stable team of mature developers who work reasonable hours. You test the hell out of the software to the point a single bug in a test is reason to redo the software. You run the software on four identical computers and make sure they all agree.
Then you hire another entire team to write code that does the same thing, but otherwise has no contact with the first team. That software runs on a fifth computer that takes over if something happens to the other four.
NASA investigators have determined that a software update performed in June of 2006 may have doomed the 10-year-old spacecraft. Apparently the software error caused the solar arrays to drive against a mechanical stop which then forced the spacecraft into safe mode.
Glad i'm not the programmer who came up with that bit of code! Their next performace review is going to be _lots_ of fun!
Actually, a subcontractor will blame another subcontractor for the fault and fighting will break out. NASA will keep peace among the subcontractors by blaming a hacker for mistaking the update as a patch for the Metal Gear Solid vidoe game, and vows not create any acronyms that could be misconstrued as a video game.
Funny definition of 'safe mode'. I'd get the main antenna pointing at the earth, the battery radiator pointing away from the sun, and the computer going 'what do I do know, smarty earthlings?' and waiting for a command.
Maybe NASA's 'safe mode' just put 'safe mode' in the corners of all the returned images and did them in 8-bit colour...
Be careful not to place too much of the blame on us programmers. Most of these crazy "business logic" equations were created by some math genius in another department. Since most of these equations mean nothing to programmers, we make sure we're typing them in correctly, since there's no way we would ever recognize any type of mistake. Most of the time the problem lies with the math guy, who was too lazy to carry a remainder, or who thought the equation was good enough being precise to four decimal places.
In other disciplines, the engineers ARE math guys. Face it, compared to other engineering types, software engineers and programmers are SLOPPY. This is because engineering has thousands of years worth of spectacular cork-ups with enormous death tolls to look back on, and engineering students are (I'm guessing, IANAE) shown horrific, traffic-safetyesque movies like Blood on the Protractor, Slide Rule Massacre, and London Bridge is Falling Down, Killing Litle Johnny's Entire Family.
Maybe we CS types need our own safety movies, perhaps When Buffers Attack!, Threads: Your Parallel Friends or Quagmires of Debugging DOOM?, or maybe Metric or Imperial: You Mean there's a Difference? Or maybe we need to recognize that many of us have the same awesome responsibility that other engineers do of protecting human lives from the consequences of our mistakes. I'm told that this point is hammered home in engineering schools, why not in CS departments?
CS people are math guys too, at least many of us are. That doesn't mean we necessarily have the expertise to validate aerospace control algorithms on the fly- that's why the's an entire discipline of aerospace engineers, because you can't expect all the *other* engineers to have sufficient knowledge. Things like this are built as teams- and team members have to make certain assumptions about the accuracy of the other team members' work. Those algorithms should have been validated before even being handed off
In other disciplines, the engineers ARE math guys. Face it, compared to other engineering types, software engineers and programmers are SLOPPY. This is because engineering has thousands of years worth of spectacular cork-ups with enormous death tolls to look back on, and engineering students are (I'm guessing, IANAE) shown horrific, traffic-safetyesque movies like Blood on the Protractor, Slide Rule Massacre, and London Bridge is Falling Down, Killing Litle Johnny's Entire Family.
Engineering and applied mathematics are much more demanding than computer programming. Sure, one could argue that "computer science is math too", but my experience is that CS majors don't graduate with a strong math background. And even if they did once know some calculus and linear algebra, they were never required to apply it like an EE or Applied Math person would.
So while you could find a rigorous programmer or software engineer (and I use the term "software engineer" very loosely, because few individuals actually fit that description), it's often a lot easier to look for an engineer or applied mathematician with good programming skills. Their math and physics is usually significantly stronger, and they actually understand what they're programming.
To put the shoe on the other foot, have you ever seen software written by people who aren't programmers? Uck. The code is usually a nightmare. Things like: "Well, here we're using the global "qzv" as a loop variable, but over here we'll use it to mean how many widgets we're looking at, and over here, it's our exit condition. Oh, and we'll set it to '5' over here for no discernable reason. Now, here's where we've cut and pasted the code 15 times so that we could change one variable's type (instead of usi
We're never going to improve as long as people insist on comparing software development to building bridges, i.e. a more sophisticated understanding of the problem is needed. In software, once you have a program for a bridge you can make a billion bridges, all alike or customized by certain parameters, just by running the program. So being "able to build the same damn bridge 100 times" doesn't get you anywhere. Making it better and safer each time? That's another story, and once again, the comparison t
It's not like the only problems with air and space vehicles have been caused by coding errors, I'm sure engineering has done fairly well for it self too.
Like the F16 thing. Let's not forget that the shuttle has NEVER been in space during a new-years. It is untested (at least in space) and they are not positive what will happen. That's why they were worried in December, they didn't want bad weather to force the shuttle to stay in space during the transition.
I wouldn't call it a failure of Computer Science; it's a QA failure without a doubt. Mistakes happen when you code. Sure, you try to minimize them but even the most carefully designed code can't be guaranteed to be 100% error free. That's why you employ, presumably, a top-notch QA team to check and recheck, testing your "perfect" code in ways that perhaps you never even considered.
This is what you would expect in a terrestrial application. When the platform that your code is going to run on isn't bound to th
For the uninformed, QA = Quality Assurance. A must-have for any self-respecting software model.
NASA has got it rough, has since the mid 70s. Their wildest successes are regarded as routine and hardly noticed by the public eye. Their failures, on the other hand, are spun to be the worst disasters in human history. Granted, when shuttles explode and people die, it's reasonable that the public be concerned. But it seems to me that for every 20 great things that NASA accomplishes, the media picks 1 failure (and sometimes blows that failure out of proportion) to rile the masses into a furious frenzy calling for the dissolution of NASA.
The F-16 didn't "bounce off the equator". Before it ever flew, in simulation the computer flipped the plane over when it crossed the equator due to a bug that incorrectly handled southern lattitudes. Additionally, since the computer "flip" happened instantaneously, and the f-16 can roll at much higher G forces than the pilot can take, the flip would have killed the pilot (and the F-16 would have happily continued on its way).
>Additionally, since the computer "flip" happened instantaneously, and the f-16 can roll at much higher G forces than the pilot can take, the flip would have killed the pilot.
Well your whole post is called into question due to quite a few questionable items:
It seems unlikely that the lattitude would enter at all into any calculation of roll attitude. If so, it's more than a "bug", it's a basic design mistake.
The F-16 does have a high roll rate, about 320 degrees per second, but since the pilot is
And don't forget the Mars Climate Orbiter "Dirt Dart" mission (http://en.wikipedia.org/wiki/Mars_Climate_Orbiter ). Okay the operators helped by plugging in the wrong units but neither did the software catch the discrepancy in the values.
The systems aboard the spacecraft were not able to reconcile the two systems of measurement, resulting in the navigation error.
Operator error but it would be interesting to figure in the number of accidents that the software could have prevented the operator from ente
Okay the operators helped by plugging in the wrong units but neither did the software catch the discrepancy in the values.
"On two occasions I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
Some expert is always trumpeting the fact that "Johnny can't program," to which many of us roll our eyes and go back to coding. But could this be a sign that the quality of the help NASA is hiring is such that these kinds of mistakes are now rampant? I mean, this could have been avoided if the code had been tested out on a full-scale mock-up of the machine, to verify that it did what it was supposed to do, before ever sending the commands to the actual machine. If anything, it's a QA failure.
In all fairness, writing code for a spacecraft is a lot harder than most of our Earthbound coding projects. These are custom-built machines running one-of-a-kind hardware; one can simulate components independently but it's very difficult to figure out how the hardware is going to behave up there in the vacuum.
For example, consider the one function of maintaining orientation. Most spacecraft use telescopes that look for star reference points. They look for particular star configurations and use microthrusters or gyroscopes to adjust their orientation. Imagine what it would take to simulate this: a zero-gravity vacuum with a realistic star-field at focus=infinity. Any laboratory mock up is going to cost a lot more than launching a new spacecraft. And that's just one subsystem.
Software upgrades at NASA go through a really rigorous quality control regimen, often requiring programmers to justify _individual_lines_ of their code to a review committee. Even then they usually won't patch noncritical bugs until the primary mission is completed.
I think your point is a good one. And the key lesson is not that NASA QA sucks, it's that programming for spacecraft is _tough_. I know they are constantly investigating new ways (like more standardization, code re-use, and formal verification procedures) of improving software reliability.
I guess those things happen. But at least it wasn't an error converting units, like the other Mars spacecraft that was lost. That is just incredibly stupid. Glad I'm not the "engineer" who wasted thousands of man-years and hundreds of millions of taxpayers' dollars because I was too stupid or lazy to convert between meters and feet.
On a positive note, it has provided me an instructive example for when I help my teenagers with their math homework. If they say it's "almost" correct, I tell them that the guy who screwed up the Mars mission probably said the same thing.
It wasn't one engineer. It was a team effort. And it wasn't a very simple matter of "forgetting". Several factors combined, including re-use of code from the MGS mission (a conversion factor was in the old code, but not recognized when the code was adapted for the doomed MCO) and budget constraints that limited pre-flight testing (so bug was missed...and in fact might have still been missed even with more testing). The effects of the bug were also subtle enough that 3 minor main engine firings were conducted without enough error showing up to reveal the problem. It wasn't until the long orbital insertion firing that the error in the trajectory became noticeable, and by then it was too late. The team's first clue something was wrong was when the spacecraft didn't radio home after the engine burn.
The details are really convoluted, but the Wikipedia page [wikipedia.org] on the mission has a decent write up explaining how the mistake was made, with additional resources cited. The PDF paper giving a perspective from the MCO team is particularly revealing, if you've got some time on your hands.
Surely it can still function on its solar arrays when its on the daylight side of the planet? Or would it drift too much out of alignment when in the dark? Or is there some other issue?
Don't believe it (Score:5, Funny)
Its most likely the Martian automated defense system setup just before we sent a probe and destroyed their civilisation [slashdot.org].
Should have used Gentoo!! (Score:2)
Re:Should have used Gentoo!! (Score:5, Insightful)
No sandbox can avoid the fact that one test was missing.
Parent
Re: (Score:2)
Re: (Score:2)
What you need to do is hold back on producing all those "fun" bugs that we all introduce into systems until you've the reputation as one of the best coders in the world, then go work for NASA and just go wild on some system that won't be used until it's in deep space and you're off working for Google, having destroyed the paper trail.
Re: (Score:2)
Where's K'Breel? (Score:3, Insightful)
Battery (Score:5, Funny)
a Technical solution I see: (Score:3, Insightful)
What is Microsoft wrote it? (Score:5, Interesting)
Re: (Score:3, Informative)
Re: (Score:2)
Re:What is Microsoft wrote it? (Score:5, Funny)
Parent
Luxury! (Score:5, Funny)
Or at least, that's how I remember it...
Parent
Re: (Score:3, Insightful)
Why don't all computers use just a single configuration (peripherals, cards, interfaces)?
The purpose of an operating system is so much wider than what the Mars Global Surveyor had to do.
Re:What is Microsoft wrote it? (Score:5, Insightful)
That said, you could get software written to this level of perfection if you wanted. It's easy- follow the space shuttle's team's example. You have a stable team of mature developers who work reasonable hours. You test the hell out of the software to the point a single bug in a test is reason to redo the software. You run the software on four identical computers and make sure they all agree.
Then you hire another entire team to write code that does the same thing, but otherwise has no contact with the first team. That software runs on a fifth computer that takes over if something happens to the other four.
Willing to pay for that?
Parent
*phew* (Score:5, Funny)
Glad i'm not the programmer who came up with that bit of code! Their next performace review is going to be _lots_ of fun!
Re: (Score:2)
Re: (Score:2)
"Safe" mode? (Score:5, Funny)
Maybe NASA's 'safe mode' just put 'safe mode' in the corners of all the returned images and did them in 8-bit colour...
Bits (Score:2)
YACCS -Yet Another Computer Corkup in Space (Score:5, Informative)
Aero and space are very unforgiving of human coding errors.
Re: (Score:2, Interesting)
Re:YACCS -Yet Another Computer Corkup in Space (Score:5, Insightful)
Maybe we CS types need our own safety movies, perhaps When Buffers Attack!, Threads: Your Parallel Friends or Quagmires of Debugging DOOM?, or maybe Metric or Imperial: You Mean there's a Difference? Or maybe we need to recognize that many of us have the same awesome responsibility that other engineers do of protecting human lives from the consequences of our mistakes. I'm told that this point is hammered home in engineering schools, why not in CS departments?
Parent
Re:YACCS -Yet Another Computer Corkup in Space (Score:4, Funny)
Parent
Re: (Score:3, Insightful)
Things like this are built as teams- and team members have to make certain assumptions about the accuracy of the other team members' work. Those algorithms should have been validated before even being handed off
Re:YACCS -Yet Another Computer Corkup in Space (Score:4, Insightful)
Engineering and applied mathematics are much more demanding than computer programming. Sure, one could argue that "computer science is math too", but my experience is that CS majors don't graduate with a strong math background. And even if they did once know some calculus and linear algebra, they were never required to apply it like an EE or Applied Math person would.
So while you could find a rigorous programmer or software engineer (and I use the term "software engineer" very loosely, because few individuals actually fit that description), it's often a lot easier to look for an engineer or applied mathematician with good programming skills. Their math and physics is usually significantly stronger, and they actually understand what they're programming.
Parent
Re: (Score:3, Insightful)
"Well, here we're using the global "qzv" as a loop variable, but over here we'll use it to mean how many widgets we're looking at, and over here, it's our exit condition. Oh, and we'll set it to '5' over here for no discernable reason. Now, here's where we've cut and pasted the code 15 times so that we could change one variable's type (instead of usi
Re: (Score:3, Insightful)
We're never going to improve as long as people insist on comparing software development to building bridges, i.e. a more sophisticated understanding of the problem is needed. In software, once you have a program for a bridge you can make a billion bridges, all alike or customized by certain parameters, just by running the program. So being "able to build the same damn bridge 100 times" doesn't get you anywhere. Making it better and safer each time? That's another story, and once again, the comparison t
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Insightful)
Mistakes happen when you code. Sure, you try to minimize them but even the most carefully designed code can't be guaranteed to be 100% error free. That's why you employ, presumably, a top-notch QA team to check and recheck, testing your "perfect" code in ways that perhaps you never even considered.
This is what you would expect in a terrestrial application. When the platform that your code is going to run on isn't bound to th
Re:YACCS -Yet Another Computer Corkup in Space (Score:5, Insightful)
NASA has got it rough, has since the mid 70s. Their wildest successes are regarded as routine and hardly noticed by the public eye. Their failures, on the other hand, are spun to be the worst disasters in human history. Granted, when shuttles explode and people die, it's reasonable that the public be concerned. But it seems to me that for every 20 great things that NASA accomplishes, the media picks 1 failure (and sometimes blows that failure out of proportion) to rile the masses into a furious frenzy calling for the dissolution of NASA.
Parent
Reliability compared to what? (Score:2)
Just one more example of how Computer Science isn't quite up to the reliability requirements of Space
And how many failures have happened because of an enginering mistake?
You seem to assume that there's zero failure in space for everything else, and 6 problems in.. 30 years? is some horrible record.
All information only makes sense in context. What's the failure rate of other components of the system?
Re:YACCS -Yet Another Computer Corkup in Space (Score:5, Informative)
http://portal.acm.org/ft_gateway.cfm?id=163293&ty
Parent
Re: (Score:3, Informative)
Well your whole post is called into question due to quite a few questionable items:
Re: (Score:2)
And don't forget the Mars Climate Orbiter "Dirt Dart" mission (http://en.wikipedia.org/wiki/Mars_Climate_Orbite r ). Okay the operators helped by plugging in the wrong units but neither did the software catch the discrepancy in the values.
The systems aboard the spacecraft were not able to reconcile the two systems of measurement, resulting in the navigation error.
Operator error but it would be interesting to figure in the number of accidents that the software could have prevented the operator from ente
Re: (Score:3, Insightful)
"On two occasions I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
Plus ça change, plus c'est la même chose.
MGS spaceship? (Score:2)
MGS? (Score:2)
We hardware types always blame software (Score:2)
Pilot said.... (Score:2, Funny)
Is this a sign? (Score:5, Insightful)
Some expert is always trumpeting the fact that "Johnny can't program," to which many of us roll our eyes and go back to coding. But could this be a sign that the quality of the help NASA is hiring is such that these kinds of mistakes are now rampant? I mean, this could have been avoided if the code had been tested out on a full-scale mock-up of the machine, to verify that it did what it was supposed to do, before ever sending the commands to the actual machine. If anything, it's a QA failure.
Re:Is this a sign? (Score:5, Insightful)
Parent
Better than a metric-English conversion error (Score:3, Insightful)
On a positive note, it has provided me an instructive example for when I help my teenagers with their math homework. If they say it's "almost" correct, I tell them that the guy who screwed up the Mars mission probably said the same thing.
-ccm
Re: (Score:2, Insightful)
KFG
Re:Better than a metric-English conversion error (Score:5, Informative)
The details are really convoluted, but the Wikipedia page [wikipedia.org] on the mission has a decent write up explaining how the mistake was made, with additional resources cited. The PDF paper giving a perspective from the MCO team is particularly revealing, if you've got some time on your hands.
Parent
So what if the battery is dead? (Score:2)
Re: (Score:3, Insightful)
zing! (Score:2, Funny)
[quote]at least if something went wrong some guy at nasa could tell his grand kids that he bricked something from ~140 million miles away.[/quote]
http://slashdot.org/comments.pl?sid=214508&cid=17