Forgot your password?
typodupeerror
NASA Mars Science

Upgrading Software From 350 Million Miles Away 228

Posted by samzenpus
from the distant-support dept.
CWmike writes "Picture doing a remote software upgrade. Now picture doing it when the machine you're upgrading is a robotic rover sitting 350 million miles away, on the surface of Mars. That's what a team of programmers and engineers at NASA are dealing with as they get ready to download a new version of the flight software on the Mars rover Curiosity, which landed safely on the Red Planet earlier this week. 'We need to take a whole series of steps to make that software active. You have to imagine that if something goes wrong with this, it could be the last time you hear from the rover,' said Steve Scandore, a senior flight software engineer at NASA's Jet Propulsion Laboratory. 'It has to work,' he told Computerworld. 'You don't' want to be known as the guy doing the last activity on the rover before you lose contact.'"
This discussion has been archived. No new comments can be posted.

Upgrading Software From 350 Million Miles Away

Comments Filter:
  • by YesIAmAScript (886271) on Friday August 10, 2012 @03:09AM (#40942705)

    It is a difficult task. While NASA has don'e a lot better than most of us programmers ever have, they have made mistakes in updating from Earth to Mars before.

    http://en.wikipedia.org/wiki/Mars_Global_Surveyor#Loss_of_contact [wikipedia.org]

    • by Taco Cowboy (5327) on Friday August 10, 2012 @03:27AM (#40942835) Journal

      That is why I do not understand why the NASA engineers want to take such a risk

      Unless it is a totally fatal software bug - that is, if they do not upgrade the software, the Curiosity rover gonna be bricked - I do not think taking the risk of bricking the rover for a regular software upgrade is worth the danger of bricking the rover, which is, as TFA has stated, 350 millions miles away
       

      • by Anonymous Coward on Friday August 10, 2012 @03:50AM (#40943013)

        99% of brickings are the result of people doing stuff that the manufacturer did not intend for you to do, on devices where important design details were hidden for commercial reasons.

        This is unlikely (one would hope) to be the case here.

        • by K. S. Kyosuke (729550) on Friday August 10, 2012 @04:55AM (#40943381)

          99% of brickings are the result of people doing stuff that the manufacturer did not intend for you to do

          In that case, that should happen with deep space probes quite a lot.

          • by RockDoctor (15477)

            99% of brickings are the result of people doing stuff that the manufacturer did not intend for you to do

            In that case, that should happen with deep space probes quite a lot.

            ... or it would do, if the manufacturers and the users weren't the same group. Or, for the likes of NASA, the manufacturers of the flight hardware computers and the manufacturers of the flight software weren't two groups of the same organisation, both of whom would take equal accountability for a failure like this. (And probably work in

        • Re: (Score:2, Funny)

          by Anonymous Coward

          pff worst case scenario : they send him over to mars to jtag the rover by hand...

      • by hcs_$reboot (1536101) on Friday August 10, 2012 @04:23AM (#40943213)

        why the NASA engineers want to take such a risk

        Similar to some devices here on Earth, the rover should have an automatic revert solution. For instance, a non-updatable software running on a separate processor detects specific conditions (like no signal from Earth for a while) and flashes back the updatable software to its original version when that condition occurs.

        • by cnettel (836611) on Friday August 10, 2012 @04:35AM (#40943279)

          why the NASA engineers want to take such a risk

          Similar to some devices here on Earth, the rover should have an automatic revert solution. For instance, a non-updatable software running on a separate processor detects specific conditions (like no signal from Earth for a while) and flashes back the updatable software to its original version when that condition occurs.

          Such things tend to be present, but how many times have they tested the automatic revert in actual conditions? An alternative codepath is always a risk.

          Updating the software can have great advantages. Only a slightly more reliable connection would allow vast amounts of more science to be done. Adapting the algorithms for autonomous functions such as simple navigation or sample processing also makes a great difference when your lag time for a single command is measured in terms of minutes and you don't even have that level of "real-time" access most of the time.

          • by Bigby (659157) on Friday August 10, 2012 @10:20AM (#40945581)

            I think it is safe to assume that they purposely bricked the rover (or test rover) before the mission. And made sure it played out as the GP stated. And that they did this many different ways.

            • by Ruie (30480)

              I think it is safe to assume that they purposely bricked the rover (or test rover) before the mission. And made sure it played out as the GP stated. And that they did this many different ways.

              Ideally - yes. In practice, they have limited funds and lots of deadlines.

              If they had lots of time to debug it, there would be no need to upload new software.

        • Haha, I wrote pretty much the same thing, at about the same time. See below.
          • I think most of us thought it. That probably means that NASA thought it too. Unless they were really against doing such a thing to save space/weight, but I think a few extra grams and square inches to have a recovery partition is definitely worth it, considering bricking the thing means you just wasted several billion dollars..

        • by Confusador (1783468) on Friday August 10, 2012 @08:29AM (#40944439)

          They do indeed have systems like that, if you're interested it's worth looking into how they dealt with the Sol 18 Anomaly on Spirit. Of particular note is the "Shutdown Dammit" command that they used to override everything else the rover was doing so it would stop wasting battery overnight.

          Seeing as they were able to update the software on a device that wouldn't even finish booting, I imagine the procedures for doing it on a functioning device are pretty robust, even if they're still nailbiting.

        • by DrXym (126579) on Friday August 10, 2012 @01:03PM (#40947845)

          Similar to some devices here on Earth, the rover should have an automatic revert solution.

          It does. Scientists put a small switch in at the back which you hold down while powering it up and it will reset itself.

      • by Jane Q. Public (1010737) on Friday August 10, 2012 @04:34AM (#40943277)

        "I do not think taking the risk of bricking the rover for a regular software upgrade is worth the danger of bricking the rover..."

        I guess it all depends on on (A) what the perceived value of the upgrade is, versus (B) the perceived risk.

        It's probably a safe bet that they learned from the Surveyor issue, and built in better tests and safeguards. I imagine -- although I don't really know -- that they have implemented something like the "rolling upgrades" that are common now, which allow processes to replaced on the fly one at a time, without reboot, and with a failsafe revert that runs at a higher level than any of those processes if anything goes wrong.

        It isn't like Windows, in which just about every time you install or upgrade something you have to make all the changes then "reboot". They get done one at a time, and they are tested individually after they are made.

        It sounds complicated but conceptually it's pretty simple: you have a top-layer monitor program program that accepts commands to replace lower-level processes. All it needs to be pretty "fail-safe" is to wait for a specified period of time for an "okay" signal from Ground Control. If it doesn't receive one in the specified time, it automatically reverts the process back to the old version. It's a little more involved than that, but that's the idea.

        Lots of software does that now. A lot has improved since 1996.

        • as long as it isn't HP writing the installer I'm ok with it... Installing printer... error... rolling back... installing.. error... ad infinitum
      • Yeah, but maybe the new JellyBean will be totally awesome!

      • by coofercat (719737)

        I imagine it's a lot easier to change the software than it is to change the hardware. I have no idea what kit the rover has in it, but since my phone camera used to take bad pictures until a software update came along, I should think Nasa probably want to upgrade the software in their cameras in preference to biking a new camera out to Mars. I seem to remember that the drill may contaminate samples with teflon or something - that being the case, I'm sure they've got a fancy filter than can remove most of th

      • Re: (Score:3, Informative)

        by necro81 (917438)
        In some cases, the software loaded on the device is not suited to the task the engineers want it to do. TFA mentions that the software on the device now is geared towards interplanetary cruise, EDL, and some very basic on-the-surface tasks. If they actually want the rover to do what they've sent it there to do, they need to perform the upgrade. Why not have the entire suite of mission software on the rover when it launches? Perhaps they hadn't gotten around to coding/testing the on-the-surface software
      • by gsgriffin (1195771) on Friday August 10, 2012 @09:23AM (#40944891)
        Probably concerned that their virus software is now out of date after the long journey.
      • by mcgrew (92797) *

        They've upgraded software on the other two rovers, as well as probes even farther away. I doubt there's any reasonable chance they'll brick it.

      • It is not such a big risk and it has been done many times before with all kinds of space crafts. And you should also realize that many safety precautions has been build into the system. It is definitely not like doing a OS update on a PC. I presume that in case something goes wrong, the rover will get into some kind of safe mode sooner or earlier, allowing to establish communication again. Safe mode communication is at a very slow speed and it could take some time to establish contact again, but in many cas
      • by pingbak (33924)

        No, the likelihood of getting bricked is really small, although the likelihood of misaligned or damaged equipment failure is much greater.

        "Bricking" is really small because there is always a known, good image that preceded the update. In the case of a failure, these spacecraft go into a "safe hold" mode (there are actually several different safe hold levels). The lowest safe hold level ensures that the operator always has access to a low-level monitor. This monitor allows the operator to select which image

    • Oblig. (Score:5, Funny)

      by AliasMarlowe (1042386) on Friday August 10, 2012 @03:29AM (#40942849) Journal

      So what's their problem? Just tell a sysadmin [xkcd.com] to fix it.

    • by Solandri (704621)
      Also Viking 1: http://en.wikipedia.org/wiki/Viking_1#Lander [wikipedia.org]
    • by kasperd (592156) on Friday August 10, 2012 @04:42AM (#40943329) Homepage Journal

      they have made mistakes in updating from Earth to Mars before.

      Sounds like it was not just a software update gone wrong but rather some mechanical problem which they were trying to work around. It was nothing like the usual bricking problem, where a firmware update overwrites code which is needed to perform future firmware updates.

      The rovers have several mechanisms to make it safer to update firmware remotely. But ultimately a combination of multiple unfortunate events can still lead to the loss of a rover. And one of those events may have been human error. From the description it sounds like mechanical problems with the solar panel, combined with two cases of human error in coordination of updates, another case of human error trying to correct the previous human errors, an unfortunate condition triggering a latent problem introduced by previous errors, and finally ending up in a position causing the battery to overheat, and loss of power being the ultimate reason it was impossible to adjust the previous mistakes.

    • by gagol (583737) on Friday August 10, 2012 @04:49AM (#40943357)
      That is probably why a team of 100 software engineers issues about 1000 commands per day for the rover. My guess is a lot of the work is triple checking everything before they upload an update. There is just no room for error in this situation.
    • by wmac1 (2478314)

      There are 2 separate computers on the board. Perhaps they upgrade one of them and after it worked correctly they transfer control to it and upgrade the other one?

  • by ronhip (465417) on Friday August 10, 2012 @03:10AM (#40942711)

    The spacecraft TRAVELLED 350 million miles to get there, but as of tonight, Mars is only about 157.5 million miles from Earth.

  • by Anonymous Coward

    Working in remote smart metering we have a similar problem, where you can brick meters if the signal drops at the wrong place, or firmware doesn't fit the hardware right.

  • Wow (Score:5, Insightful)

    by undulato (2146486) on Friday August 10, 2012 @03:15AM (#40942747) Homepage
    NASA doing a software upgrade is not big news. This is going to be phenomenally safe. Much scarier doing software upgrades on millions of unknown hardware configurations globally than on one totally locked down platform no matter what distance or cost is involved.
    • Agree, NASA has done a complete upgrade on the previous rover. This isn't new stuff and it has been tested, the procedure is well known. Well, yes, someone may do stupid thing at the wrong time, however, the main difference is the speed of transfer and the delay between transmission and confirmation everything went fine. The environment is well controlled and I do not doubt there is fallback mechanisms in place. So, I'm sorry, on this one I am not really impressed by the NASA team.
      • by jeko (179919) on Friday August 10, 2012 @04:34AM (#40943273)

        Get a 10-foot 4X4 piece of lumber. Drop it flat on the ground. Walk from one end to the other like a balance beam. I'll bet you can do it. I'll bet you can do it blindfolded, walking backward. I'll bet you can do it reciting the alphabet backward. I'll bet you could do it drunk.

        Take that same 4X4, suspend it 20 stories in the air between a couple of cranes. Put a bunch of razor sharp, rotating propellers on the ground beneath it. Intersperse the propellers with oil drillbits pointed up, not down for once. Have a bunch of trained turkey vultures flying around to watch you fall. Take your wife, kids and your momma, put a gun in their mouths while the Joker cackles that when you fall, he's gonna blow their heads off. Bring in the television cameras and monitors so the whole World can watch and you can watch them watch. Have some intern read the tweets and comments sections about your plight over the loudspeakers.

        Now, there are a few ice-blooded "Licensed to Kill" Double-O men who could keep it together and walk that beam under that kind of pressure. Mary Lou Retton and Nadia could, no doubt. I seriously doubt I could.

        Is it a big deal to do a software upgrade under such tightly controlled conditions? Not really. But try doing that software upgrade when billions of dollars and your career is on the line, with the whole world watching. The guy who screws that up is gonna be a punchline and a byword for a few decades, a real Wilson if you've read that book. :-) You'll be known as the guy who screwed up Mars.

        Tell me there wouldn't be maybe one or two drops of sweat on the keyboard...

         

      • by Bigby (659157)

        I'm impressed that they found budget dollars for proper testing. Maybe they estimated the cost of failure to be $2.5b. I wish I could do that at work.

  • By pressing F8 at the "Starting Windows 95" message, and then choosing Safe Mode from the Windows 95 start-up menu.

    Following these steps will gain you ultimate FAME and FAILURE - for updating the Mars software!!!

  • Imagine how far it would have been if they had measured it in kilometers instead!

    Whoaw!

    .
    .
    -
    .
    .
    . ;)

    • by fa2k (881632)

      For those distances I just read "miles" as "kilometers". A factor of 1.6 doesn't really make a huge difference for a casual understanding.

  • by Anonymous Coward

    sudo apt-get update mars

  • It will sit there forever: "Are you sure you want to update? Yes/No"
  • If you follow "Scott Maxwell" in google plus, there are some great snippets about the landing and software. See: https://plus.google.com/u/0/112648317373638762082/posts [google.com]
  • by symes (835608) on Friday August 10, 2012 @06:01AM (#40943697) Journal

    They are bound to have a copy of Curiosity here on Earth, surely? So they should be able to thoroughly test the process first. Ok, it is not Mars and there might be issues specific to transmitting that data over such distances... but still. I'd be really surprised if this hasn't been thoroughly tried and tested.

    • by ledow (319597)

      More than that, if you design the system properly it would never be a problem.

      Watchdog timers on everything - on the hardware coming up, on the communications with Earth, etc. If you don't get a response from the timers in X seconds/minutes/days, then completely revert to the previous version of the software and try again.

      So if you upgrade the software and break the radio, in a day or so of not being able to talk to Earth, the machine should notice and revert back to the previous software. If you break th

  • But the tecnologies used in some botnets are a goot starting points.
    That'd be, call home and try to pull anything you need to do the upgrade.
    The orbiter relay should be doing the same, first.

  • a new version of the flight software on the Mars rover Curiosity

    Is anybody else thinking that any changes to the flight software is now a few days too late?

    • Yeah, the freakin summary is potty as usual.

      They aren't upgrading the flight software. They are replacing the flight software with driving around and exploring software.

  • I mean, the lag is going to be on par with SSH in to a terrestrial server with my AT&T service and cell phone.

  • People complain of 300ms of latency here on earth with their ISP. I have heard it takes 14 MINUTES for a signal round trip. Thats 840 seconds, or 840,000ms of latency. So you are not exactly programming on the fly.

    The worst part, would be that presumably there is some pretty robust simulated debuggery on earth before anything gets transmitted. However once you finally tested, confirmed, compiled, packaged etc... and press the send button. You have to wait likely an eternal excruiciating 14 minutes before yo

  • I love how everyone here is like, "Y'know, they really should have a backup software solution on the rover" or "If I was doing this, I would do this, that, and the other thing, and they're stupid for not doing that".

    An awful lot of assumptions being made about people who are probably the very top of their game. I'm going to give NASA the benefit of a doubt here: I think they wouldn't do the upgrade unless it was very beneficial, and I'd bet they're doing it in a way that has layers upon layers of safeguard

"Everything should be made as simple as possible, but not simpler." -- Albert Einstein

Working...