Forgot your password?
typodupeerror
Mars Programming Space Science

Programming Error Doomed Russian Mars Probe 276

Posted by Soulskill
from the to-infinite-loops-and-beyond dept.
astroengine writes "So it turns out U.S. radars weren't to blame for the unfortunate demise of Russia's Phobos-Grunt Mars sample return mission — it was a computer programming error that doomed the probe, a government board investigating the accident has determined." According to the Planetary Society Blog's unofficial translation and paraphrasing of the incident report, "The spacecraft computer failed when two of the chips in the electronics suffered radiation damage. (The Russians say that radiation damage is the most likely cause, but the spacecraft was still in low Earth orbit beneath the radiation belts.) Whatever triggered the chip failure, the ultimate cause was the use of non-space-qualified electronic components. When the chips failed, the on-board computer program crashed."
This discussion has been archived. No new comments can be posted.

Programming Error Doomed Russian Mars Probe

Comments Filter:
  • by LostCluster (625375) * on Tuesday February 07, 2012 @03:25PM (#38958015)

    We've got a contradictory summary here. Chip failure isn't a programming fault, it's a hardware problem. Stop confusing hardware and software you insensitive clod.

    • Re: (Score:3, Insightful)

      by Anonymous Coward
      Obviously the error handling routine was poorly written.
      • by Anonymous Coward on Tuesday February 07, 2012 @03:31PM (#38958143)

        sure, it missed:

        if(cpu_melted)
              abort();

      • "Cosmic rays?"
        "That's a software problem...

        They're lucky those chips they bought from China weren't made of lead, or contain deadly melamine!!!

        • by sconeu (64226) on Tuesday February 07, 2012 @04:17PM (#38958847) Homepage Journal

          You laugh, but how many of you low level guys had to work around buggy hardware?

          I once sent a memo to my boss that I was doing the equivalent of "working around a burnt out lightbulb in software".

          E.g.: How many hardware guys does it take to change a lightbulb? None, we'll just have the software work around it.

          • by mevets (322601) on Tuesday February 07, 2012 @04:50PM (#38959293)

            Try this one on your hardware guys:
            "The main purpose of software is to make hardware reliable".

            Drives them nuts...

          • Not even necessarily low level. I once had a weird intermittent problem in a PHP driven web system. After a couple of weeks of diagnosing (largely trying to find a case the could more-or-less reliably tickle the bug), it turned out to be an interaction of a bug in the Redhat version of that day (2001) with a bug in the particular CPU we were using. PHP code just happened to trigger it under certain conditions. Since the box was at Level 3, we had to drive an hour down there and replace the machine.

            And long ago I worked on Perq workstations, which had a stack-machine CPU (the CPU was a 15x15 inch board filled with TTL). The expression stack was four chips. The system was designed around the chip spec - NEVER DO THAT!!! Chips can not be depended to go at exactly the design spec - some are slow, some are fast. As a result, every CPU had to be tested at installation with those four chips inserted in different locations, essentially in order of speed. If a fast one came after a slow one in the slots, the CPU would barf. Basically someone just kept swapping chips around until it worked.

            We were just discussing some of the remarkable repairs done in software to accommodate problems in various interplanetary probes - truly amazing stuff.

        • by jd2112 (1535857)

          "Cosmic rays?" "That's a software problem...

          They're lucky those chips they bought from China weren't made of lead, or contain deadly melamine!!!

          If they were made of lead they might have blocked enough radiation to prevent them from crashing.

      • by icebike (68054) * on Tuesday February 07, 2012 @03:54PM (#38958521)

        Obviously the error handling routine was poorly written.

        I'll assume your tongue was firmly planted in your cheek, and suggest a +1 Funny mod.

        But on the chance you were serious, depending on where that chip was, it may have been beyond something manageable by software.

        A chip in a power controller could take down any or all of the processor components, or render access to control circuits impossible.

        The linked article also states

        Everything was working well with the spacecraft immediately after launch, including deployment of the solar panels, until the command to start the engines was issued. When that did not happen, the spacecraft went into a safe mode, keeping the solar panels pointed to the Sun to maintain power.

        How many times do you supposed they actually tested engine start IN THE SPACE CRAFT? I'm guessing ZERO.

        non-space qualified parts being used in some of the electronics circuits. This is a design failure by the spacecraft engineers that might have been caught had they performed adequate component and system testing prior to flight. But they did not.

        So design failure, due to radiation, prior to the craft getting near the strongest radiation belts. Unbelievable. Occam would be skeptical.

        This sounds to me like some on-board internal source of radiation, or induction, or simple overload, fried a chip somewhere in some un-specified circuitry, most probably in the engine controls. This seems far more likely than an external radiation source given the shielding the physical design would provide.

        I doubt space qualification made any difference at all. The window for space radiation in the brief time it was operational was small.
        Rather I suspect under-spec parts, over voltage or high current draw, or internal shielding oversights.

        • by Rakishi (759894)

          How many times do you supposed they actually tested engine start IN THE SPACE CRAFT? I'm guessing ZERO.

          I'm sure they tested the engine multiple times. I'd figure the stress of the launch (vibrations, etc, etc.) causes something to fail either due to shoddy construction or small debris falling onto something.

          I doubt space qualification made any difference at all. The window for space radiation in the brief time it was operational was small.

          Exactly. I doubt all those laptops on the ISS are radiation hardened but they last quiet a while anyway.

          • by icebike (68054) *

            How many times do you supposed they actually tested engine start IN THE SPACE CRAFT? I'm guessing ZERO.

            I'm sure they tested the engine multiple times. I'd figure the stress of the launch (vibrations, etc, etc.) causes something to fail either due to shoddy construction or small debris falling onto something.

            I'm sure they tested the engines too. Its probably a tried and true engine. The Russians tend to make very good motors.

            But I seriously doubt they tested it in the space craft using the space craft's wiring harness. They used the harness on the test bed platform.

          • by ChrisMaple (607946) on Tuesday February 07, 2012 @06:04PM (#38960239)

            There are many aspects to radiation hardness. Radiation can flip one or more bits, resulting in bad data or program crash. Radiation can cause latchup, which will last until power is cycled; if the design is bad, latchup can fry a part. Rad hard parts are designed to be resistant to latchup. Really bad radiation can damage a part that isn't even powered.

            A laptop can live through bit flips, and with luck it can live through latchup, and be functional after power cycling. Spacecraft control generally has to be always on; power cycling in not an option. Thus the design requirements for spacecraft control must be much stricter.

    • by Cochonou (576531) on Tuesday February 07, 2012 @03:37PM (#38958221) Homepage
      Well... if you read TFA (or actually the first TFA linked), it is clearly written:
      In a report to be presented to Russian Deputy Prime Minister Dmitry Rogozin on Tuesday, investigators concluded that the primary cause of the failure was "a programming error which led to a simultaneous reboot of two working channels of an onboard computer [...] Likewise, cosmic rays and/or defective electronics are not the leading suspects behind Phobos-Grunt’s demise.
      The summary is clearly bolting together two contradicting reports.
    • by Rary (566291)

      The second link makes the following claim:

      In a report to be presented to Russian Deputy Prime Minister Dmitry Rogozin on Tuesday, investigators concluded that the primary cause of the failure was "a programming error which led to a simultaneous reboot of two working channels of an onboard computer," the Russian state-owned news agency RIA Novosti reported.

      However, the third link says nothing of the sort. It sounds like TFS is just a mishmash of conflicting theories from different articles.

      • by Rary (566291) on Tuesday February 07, 2012 @03:45PM (#38958359)

        To follow up, the article saying that it was a chip failure is dated yesterday, while the article claiming it was a programming failure is dated today. Presumably, this is new information to shoot down the previous claims, but TFS (in typical Slashdot "editorial" style) fails to actually make that distinction, and puts both claims together as part of a single summary.

    • Chip failure, but it was a software error that lead to not handling the chip failure gracefully. Space qualified stuff has to be much more redundant and capable of handing failures of multiple components.

      • by 0123456 (636235)

        A while back I read some interesting discussions between satellite engineers about the tradeoffs between space qualified and not space qualified chips. From what I remember you gain resistance to radiation, but lose in other areas such as resistance to physical damage (e.g. a solder joint coming loose due to launch vibrations) because they're so far behind the state of the art that you may have to put a lot more chips on the same circuit board.

        So it doesn't seem a clear-cut choice... rebooting the computer

        • by Grishnakh (216268)

          I'm not a satellite engineer, but wouldn't it be easy enough to just install a lead shield around the PCB to protect from most radiation? As long as the shield's not too thick, it shouldn't add too much weight, especially compared to using older-technology chips that'll take up more board space.

          • by K. S. Kyosuke (729550) on Tuesday February 07, 2012 @05:06PM (#38959515)

            I'm not a satellite engineer, but wouldn't it be easy enough to just install a lead shield around the PCB to protect from most radiation? As long as the shield's not too thick, it shouldn't add too much weight, especially compared to using older-technology chips that'll take up more board space.

            Well, that depends. Even on Earth's surface, we have to use ECC in more demanding application. In LEO, you lose the protection of the atmosphere but you still have Earth's rather strong and large magnetosphere. But this was an interplanetary probe. Once you get out of the radiation belts, interstellar and intergalactic particles start hitting you. You can't protect from those with a lead shield of any reasonable size. Pretty much the only way is simply to make the chip simple, rugged and design it with components (transistors) large enough that a particle flying through won't bother you much. Or add redudnancy. Or both, if possible (that's the usual case).

    • by icebike (68054) *

      The second link in summary leads to an article that is internally contradictory. That page from Discovery News is all over the place.
      Which is not surprising given the bio of the author [discovery.com]:

      Klotz came to Brevard County, Fla. (aka The Space Coast) as a copy editor for the local paper 24 years ago. She switched to writing because it was obvious the reporters were having way more fun than the editors for the same money. After a year or so of writing for the business section,
      Journalism major trying to wear the big girl shoes.

      The Link to the planetary society page seems much more reliable.

    • by smcdow (114828) on Tuesday February 07, 2012 @04:06PM (#38958699) Homepage

      You can't possibly call yourself a programmer if your code can't recover from a hardware fault.

    • by geekoid (135745)

      If a software failover fails, and the currently used chip fail, then it's both.

      Please, do some low lever software /hardware work before opening you mouth.
      This isn't one of your slapped together VB3 front end.

      Yeah, YOU Herd me.

    • by gatkinso (15975)

      Space rated hardware, software, and (more relevantly) firmware is designed to handle this type of problem (to the fullest extent possible).

    • Re: (Score:3, Insightful)

      by alienzed (732782)
      On the other hand, this demonstrates so aptly why they failed in the first place. "Yep, it's a software problem, because the hardware failed to run any after it was damaged."
    • by jamstar7 (694492)
      At least they didn't fuck up a meters-to-feet conversion.
    • by crutchy (1949900) on Tuesday February 07, 2012 @04:39PM (#38959169)
      to my knowledge, only the Apollo Guidance Computer has ever truly achieved hardware failure tolerance. the Apollo 11 LM radar fault overloaded the computer, but was able to continue due to restart logic built into the AGC that was able to pick up critical tasks from where they were when the computer was restarted and drop non-critical tasks, and all with a very small fraction of the capabilities of current technology (although I think from memory they were able to fit 2 transistors on a single chip!). the AGC is really a marvel of (past) engineering and computer science. the reliability problem alone would be insurmountable with today's garbage. probably part of the reason why we haven't been back there since.
  • by mehrotra.akash (1539473) on Tuesday February 07, 2012 @03:28PM (#38958075)

    the ultimate cause was the use of non-space-qualified electronic components

    Programming error?
    Perhaps in the software used to order the parts

  • headline fail (Score:3, Informative)

    by jamessnell (857336) on Tuesday February 07, 2012 @03:29PM (#38958095) Homepage
    "the ultimate cause was the use of non-space-qualified electronic component" != "programming error" hardware fail.
    • by X0563511 (793323)

      Even better... a design fail! The hardware worked (or not) as per it's specifications. It's not the hardware's fault you put it where it wasn't meant to go!

    • by geekoid (135745)

      A) Some hardware has software embedded into it, yeah shocking.

      B) Parts fail in space craft. If the software failed to detects a failed piece and roll to back up, the software has it's roll in the incident as well.

      C) If it jump to the wrong mode after the error, that's also a software error.

      I'm not saying one way or another in the specific incident. The idea that there is a hard line between all software and hardware is false, and technical people should know better.

    • by jd2112 (1535857)
      Programmer didn't include if($component.SpaceRating != TRUE) {throw "INITIALIZATION ERROR: NON SPACE RATED COMPONENT!"}
  • by cvtan (752695) on Tuesday February 07, 2012 @03:31PM (#38958145)
    How much did they save by using Radio Shack parts in a Mars probe? $5.00 even?
    • by Spykk (823586) on Tuesday February 07, 2012 @03:50PM (#38958447)
      Not even the government could save money by buying something at Radio Shack.
    • Re:So how much? (Score:4, Informative)

      by stewbee (1019450) on Tuesday February 07, 2012 @03:53PM (#38958499)
      If only. The reason ICs cost so little is that the cost is spread out over millions of parts. As my analog circuits Prof would say. "Your very first IC off the line is going to cost a million dollars. Everything else after that is free." So to buy one or two ICs that are radiation hardened is probably going to cost that much since it will most likely be custom. Now that's not to say they can't reuse some of the masks for an existing IC to make it cheaper, but It won't be that much cheaper. My guess is that they would want to redesign the part anyway if it is going to be in a radiation intense environment. The radiation could cause some weird quantum effects in the IC that might mean they want the transistors to be larger for reliability purposes. But that last part is just a guess since I am not an IC designer and thought my electronic materials class was nothing short of voodoo.

      Long story short, they probably saved more than $5 for using a COTS part, but they probably lost the probe by the part not being radiation hardened.
      • Re:So how much? (Score:4, Interesting)

        by autophile (640621) on Tuesday February 07, 2012 @04:48PM (#38959281)

        For want of a rad-hard chip, the board died.

        For want of a board, the software couldn't cope.

        For want of good software, the engine start failed.

        For want of engine start, the probe died.

        For want of a probe, the human race didn't detect the slimy aliens from Phobos and all perished in a hot and somewhat greasy fireball.

    • How much did they save by using Radio Shack parts in a Mars probe? $5.00 even?

      Based on my last visit to Radio Shack, I don't think their parts are any cheaper than the special-purpose, radiation-hardened parts they should have used.

      But when you can't wait until tomorrow for a part for your space probe, Radio Shack is convenient.

      • by yurtinus (1590157)
        I dunno, seems to me it'd be quicker just to order your parts from Digikey instead of going to Radio Shack, buying a cell phone and contract, then dismantling the phone to desolder the part you need (and hope you didn't bust the part in the process)... Sure, Radio Shack is convenient for a lot of things, as long as all of those things are cell phones and expensive Ethernet cables.
    • by systemeng (998953)
      When I worked in the test equipment industry, we had a term for the lowest grade of parts that still worked when binning components: The radio shack bin. I once built part of an emergency prototype for a test equipment cooling system with radio shack parts. The prototype was sent to Taiwan where it failed prematurely due to the marginal components. Never Again!
    • How much did they save by using Radio Shack parts in a Mars probe? $5.00 even?

      This is not the first time something like this happened to the Russians. In the 1970's, the Soviet Mars 4 [wikipedia.org] probe failed in flight. The reason? Due to cost savings, the transistors used had had their gold parts replaced with aluminium ones, which were prone to chemical degradation (a.k.a. corrosion). The Soviets then realized that they had manufactured three more probes of the same series using the same (unfit) transistors. Now what did they do? Of course they launched them! Guess what happened? Mars 5 failed

    • Re:So how much? (Score:5, Interesting)

      by jd (1658) <imipak@noSPam.yahoo.com> on Tuesday February 07, 2012 @04:40PM (#38959187) Homepage Journal

      Space Micro [spacemicro.com] doesn't list the prices of their components or systems, nor can I find any from anyone else. Honeywell [honeywellm...ronics.com] don't list their prices either. Atmel seem to have dropped out of the field. Linear [linear.com] don't list the prices for their space-hardened stuff. Don't see any for BAE [baesystems.com] either, or Intersil [intersil.com]. Empire Magnetics [empiremagnetics.com] require a lot of personal data before they give you access to even the price classification information. Not the prices, just how they're classified.

      You've got to allow for a year's worth of traveling outside of an atmosphere and then operating on Mars for the duration of the mission. This analysis of radiation for manned missions [esa.int] suggests you're looking at 3.5 mSv per day, then 20 rems per year [solarstorms.org] in most of the places of interest.

      Converting everything to rads, it's 0.1 rads per mSv and 1 rad per rem, so that's 12.75 rads to get to Mars if you assume a year-long trip, plus 20 rads for the mission, so anything with a rating of less than 32.75 rads is pretty much guaranteed to fail. However, over the course of a two years, the odds of there being a [hps.org]solar flare [nasa.gov] are not insignificant. To be safe, you want resistance to a further 400 rad. 432.75 rad is within the tolerance of most of the space-hardened components (some components can be taken up to 1000 rad, others up to 10,000). However, the cheapest space components would NOT survive. You're talking high-end on the space scale.

      I'm going to figure that the top-line components will cost 100x that of their conventional counterparts, due to the higher-level of precision and QA that are required. It might well be a good deal more. In Russia, you've also got to pay for smuggling decent-grade hardware out of the US, as all of this stuff will be under massive amounts of regulation.

      My guess is that the cuts would have saved enough that those doing the cost-cutting could buy second homes in Switzerland.

  • by invid (163714) on Tuesday February 07, 2012 @03:33PM (#38958163) Homepage
    Is it just me, or is it the responsibility of all software engineers to find the hardware problem in order to prove to people that the cause isn't software?
    • is it the responsibility of all software engineers to find the hardware problem in order to prove to people that the cause isn't software?

      Find someone else, I'm busy.

      In any case, it's usually orders of magnitude easier to blame the spec. It's written by management/users, after all...

    • by rwv (1636355)
      In my experience... hardware problems are acceptable if there's a software work-around. Special acknowledgement isn't given to software for fixing hardware bugs... it's just expected since hardware is arguably more expensive to change.
  • Components. American components, Russian Components, ALL MADE IN TAIWAN!

    http://www.imdb.com/title/tt0120591/quotes?qt=qt0459113 [imdb.com]

  • Contradictions (Score:5, Informative)

    by Aladrin (926209) on Tuesday February 07, 2012 @03:39PM (#38958265)

    The summary is so contradictory because it quotes from 2 articles, and each of them is completely different. One says that the parts were space-tested and fine, and the other says they were never space-certified and were definitely bad. The first one says instead that a software bug caused parts of the system to reboot. The second doesn't know what happened and just blames faulty hardware.

    • by mbone (558574)

      The summary is so contradictory because it quotes from 2 articles, and each of them is completely different.

      " A foolish consistency is the hobgoblin of little minds, (Emerson)

  • by kbob88 (951258) on Tuesday February 07, 2012 @03:43PM (#38958327)

    In other news, U.S. radars were not responsible for the highly confusing and contradictory summary posted this morning to a Slashdot story about Russia's Phobos-Grunt probe. A thorough investigation has determined that the story's chips should have been able to withstand the radiation received when the story was transmitted through the intertubes and routed over northern Alaska. Instead, investigators blamed a typing failure on the story editors. "A series of tests showed that the editing was lousy and sloppy, and disciplinary action will be taken on those responsible," a spokesman said.

  • In Soviet Russia probe causes programming bug!

    They have very strict security measures. It can be traumatic.

  • What's with Mars and probes? Seriously, how many have been lost either going or coming from?
  • by billcopc (196330) <vrillco@yahoo.com> on Tuesday February 07, 2012 @03:51PM (#38958477) Homepage

    Okay, we still have a respectable though dwindling community of commenters, so can we please get rid of these editors who can't even be bothered to read four lines of summary text before posting ?

    The headline and summary do not make sense. Come on, we're supposed to be nerds, aka intelligent, focused, attentive knowledge aggregators.

    the fuck is wrong with this goddamned site?! These failures are starting to make Digg look good!

    • by geekoid (135745)

      No. You are welcome to go to the times and pay for a subscription that uses actual editors.

  • by vlm (69642) on Tuesday February 07, 2012 @03:52PM (#38958479)

    Fun to read the comments here. I've done embedded stuff and you need to be defensive. You can see at a glance who here has never done defensive programming before, or embedded or safety critical programming, all blaming the hardware. There's 3 states so you got 2 bits of input and a disallowed state comes in. Deal with it, don't just curl up and die and blame the hardware designer. There's a 12 bit A/D conversion result stored in two bytes, and there's a 14 bit number found there, deal with it don't just curl up and die and blame the ... . Theres a cycle start button and an emergency stop button and both are simultaneously on. Deal with it. You reboot a mission critical (or safety critical!) CPU and a minor auxiliary input A/D doesn't initialize, do you burn the plant down in a woe is me pity party because one out of 237 sensors aren't coming on line, or do you deal with it?

    Finally radiation is a statistical phenomena. There is no such think as radiation free. If they used non-rad hardened parts, its gonna crash maybe 10000 times more often. Thats OK, you program around that, assuming you know what you're doing. Radiation hardened does not equal radiation-proof. If there was a single bit error, or a latchup on a rad-hardened unit, with a poorly programmed control system it would have failed just as well, its just that a rad hardened chip would have made it a couple orders of magnitude less likely. A shitty design that has a 1 in 20000 failure rate due to better hardware instead of 1 in 2 is still a shitty programming design, even if the odds are "good enough" that it makes it most of the time with the better hardware.

  • Baloney (Score:5, Interesting)

    by mbone (558574) on Tuesday February 07, 2012 @03:58PM (#38958573)

    What are the chances chips would fail in a 20-30 minute period just after launch but before Mars transfer orbit insertion ?

    No, I bet this was a programming error, coupled with a near total failure to test the software.

     

    • What are the chances chips would fail in a 20-30 minute period just after launch but before Mars transfer orbit insertion?

      Small, but decidedly non-zero. So I should point out that "improbable" != "impossible".

  • I read the title and I was going to make a joke forgetting a ;, or something in the like.
    But this wasn't a programming error, it was a hardware failure |:
    Did the editor even read what he wrote?
    • by Fnord666 (889225)

      Did the editor even read what he wrote?

      The editors no longer write or read anything. They just cut and paste. Submitters no longer write anything, they just copy the first paragraph or two of an article. I swear that some days all of the articles are probably just submitted by a very short perl script.

  • by walterbyrd (182728) on Tuesday February 07, 2012 @04:40PM (#38959189)

    Ripped from old David Letterman "Top Ten List"

    10. "Mars probe? What Mars probe?"
    9. Forgot to use The Club
    8. Those lying weasels at Radio Shack
    7. Too much Tang
    6. Made by G.E.
    5. Them Martians musta shot it down with a ray gun
    4. Heh, heh, heh ... Our space probe sucks -- heh, heh, heh
    3. At least we didn't blow all our money on some dork screwing around with a car phone
    2. Remember Watergate? Well, Nixon's up to his old tricks again!
    1. Space monkeys

  • by funkboy (71672) on Tuesday February 07, 2012 @06:24PM (#38960441) Homepage

    Well, if there was an RTG [wikipedia.org] onboard, then maybe the radiation damage was from inside the spacecraft.

    It seems strange to me that they'd blame radiation damage as they have a separate institution dedicated to developing rad-hard SPARC chips for space applications [wikipedia.org] that has a very successful track record.

    Question: how do they know it was radiation damage if they never heard back from the probe?

  • by Darth Hubris (26923) on Tuesday February 07, 2012 @07:10PM (#38960935)

    Who saw "Doom", "Mars", and "Phobos" and reached for your shotgun?

No amount of careful planning will ever replace dumb luck.

Working...