Forgot your password?
typodupeerror
Space Bug Wireless Networking Science Hardware

Debugging The Spirit Rover 390

Posted by timothy
from the at-a-distance dept.
icebike writes "eeTimes has a story on how the Mars Rover was essentially reprogrammed from millions of miles away. 'How do you diagnose an embedded system that has rendered itself unobservable? That was the riddle a Jet Propulsion Laboratory team had to solve when the Mars rover Spirit capped a successful landing on the Martian surface with a sequence of stunning images and then, darkness.' The outcome strikes me as an extremely Lucky Hack, and the rover could have just as likely been lost forever. Are there lessons here that we can use here on the third rock for recovery of our messed up machines which we manage from afar via ssh?"
This discussion has been archived. No new comments can be posted.

Debugging The Spirit Rover

Comments Filter:
  • Oh, sure... (Score:5, Funny)

    by inertia@yahoo.com (156602) * on Sunday February 22, 2004 @01:43AM (#8354094) Homepage Journal
    Are there lessons here that we can use here on the third rock for recovery of our messed up machines which we manage from afar via ssh?

    As a former co-worker (hi, jwalker!) used to say when people tried to draw ridiculous analogies, "It's exactly like that...only different."
    • Re:Oh, sure... (Score:5, Insightful)

      by JWSmythe (446288) * <(moc.ehtymswj) (ta) (ehtymswj)> on Sunday February 22, 2004 @03:25AM (#8354470) Homepage Journal
      It sounded like the same type questions non-technical bosses always ask about technical matters.

      "We're ordering this brand new hardware that you've never tested before. Can you guarantee it will never crash?"

      "Will this database server handle the load of our brand new project?" (without an accurate growth estimate)

      "A server 2000 miles away just went down. What happened?" (no ping, no nothing) Hmmm.. Power/NIC/CPU/CPU fan/hard disks?

      It really sounds like they did some decent advanced planning on those probes, but from other stories I read, the were shooting for 90 days of reliability, which in itself was a hard one to do. What if it turns the antenna the wrong way and looses connectivity? What if it gets hit by lightning? What if it falls in a hole? (go Beagle!)

      Sure, relate this to your web server colocated somewhere you're not. Cross your fingers, hold your breath, and hope there aren't a few fatal systems failures, or a bit of human error. I've been responsible for a bit of that in the past, but at least my equipment wasn't a few million miles away.

      • Re:Oh, sure... (Score:4, Informative)

        by FrostedWheat (172733) on Sunday February 22, 2004 @09:05AM (#8355049)
        What if it turns the antenna the wrong way and looses connectivity? What if it gets hit by lightning? What if it falls in a hole? (go Beagle!)

        There is a low gain omni-directional antenna that can be used as backup. Infact I think they use it most of the time for commands and just use the high-gain for data transfer back to Earth. Which makes sense, they never need to send large amounts of data to the rover.

        No lightning has ever been detected on Mars. Tho it's not impossible, it is very very unlikely. No proper observations of the night side of Mars has been done tho, so they may just be missing it.

        And Opportunity did fall into a hole :)
    • by alwaystheretrading (750171) on Sunday February 22, 2004 @03:28AM (#8354482)
      That must have been some feat to get the arm on the rover to press Ctrl, Alt and Delete at the same time!
    • Re:Oh, sure... (Score:4, Informative)

      by sjames (1099) on Sunday February 22, 2004 @11:22AM (#8355459) Homepage

      Actually though, it's not too bad an analogy. While Earth based servers aren't absolutely unreachable like SPirit, they are often remote, and there are expenses associated with visiting them in person.

      Various schemes now exist to help deal with that. Many boards have a small management processor (bmc, server management board, IPMI, whatever) that is used for remote diagnostics and reconfiguration when the main board won't even boot.

      Meanwhile, LinuxBIOS supports two complete BIOS images. One 'old reliable' that once working is never changed, and one that can be upgraded freely. Coupled with a watchdog card or timer, it's decently managable in the field. That work is continuing.

      Meanwhile, IBM is pushing the 'blue button' that forces a software reload from an image partition.

      In that sense, the problem is strongly analogous. Most of us will not, however, encounter the exact problem that Spirit had, though some embedded device developers just might.

  • by webmaestro (323340) * on Sunday February 22, 2004 @01:44AM (#8354095) Journal
    Man, I have a hard enough time debugging programs running on my local machine.
    • by srichand (750139) on Sunday February 22, 2004 @01:56AM (#8354168)
      In other news stories, the Microsoft Corporation decided to sue NASA, apparently since the right to crash systems was only theirs. Not to be left behind, SCO insisted that the code that caused the failure was unethically copied from their source repositories. This has indeed caused a flutter in the space communities
  • by detritus` (32392) * <awitzke@wesayso. o r g> on Sunday February 22, 2004 @01:45AM (#8354098) Homepage Journal
    I dont think i want to learn too much from this as the solution was the equivalent of rm -rf... On a side note i wonder when the 40 min ssh delay jokes will begin again
  • well (Score:4, Funny)

    by whackco (599646) on Sunday February 22, 2004 @01:45AM (#8354104) Journal
    at least it wasn't a blue screen?
  • Like this? (Score:4, Funny)

    by The Human Cow (646609) on Sunday February 22, 2004 @01:45AM (#8354106) Homepage
    man rover?
  • by Nimloth (704789) on Sunday February 22, 2004 @01:47AM (#8354114)
    I don't get it, couldn't NASA afford the on-site warranty?
    • by kfg (145172) on Sunday February 22, 2004 @02:08AM (#8354227)
      Yeah, but they thought they could save a few bucks and got the Gateway consumer version.

      "Oh, you've got the on-site warranty, huh? Ok, first thing you have to do is ship it to South Dakota. . ."

      Oh, hey, looks just like Mars.

      KFG
    • rebooting on mars... (Score:5, Interesting)

      by segment (695309) <silNO@SPAMpolitrix.org> on Sunday February 22, 2004 @02:17AM (#8354268) Homepage Journal
      Interesting reading:

      Rebooting on Mars

      By Matthew Fordahl, The Associated Press

      It's a PC user's nightmare: You're almost done with a lengthy e-mail, or about to finish a report at the office, and the computer crashes for no apparent reason. It tries to restart but never quite finishes booting. Then it crashes again. And again.

      Getting caught in such a loop is frustrating enough on Earth. But imagine what it's like when the computer is 200 million miles away on Mars. That's what mission controllers faced when the Mars rover Spirit stopped communicating last month.

      ...

      Tech support for an $820 million mission is a cautious affair. Tools to recover from and fix any problem must be built into the system before launch. The systems' behaviors need to be completely understood and predictable.

      "Luckily, during the design period, we anticipated that we might get into a situation like this," said Glenn Reeves, who oversees the software aboard the Mars rovers Sprit and Opportunity at NASA's Jet Propulsion Laboratory.

      For stability, reliability and predictability, mission designers did not bust the budget and design the hardware or software from scratch. Instead, they turned to hardware and software that's been used in space before and has a proven track record on Earth as well.

      "The advantage of using commercial software is it's well-known, and it's well deployed," said Mike Deliman, an engineer at Alameda-based Wind River Systems Inc., which made the rovers' operating system. "It has been used throughout the world in hundreds of thousands of applications."

      The operating system, VxWorks, has its roots in software developed to help Francis Ford Coppola gain more control over a film editing system. But the developers, David Wilner and Jerry Fiddler, saw a greater potential and eventually formed Wind River, named for the mountains in Wyoming. VxWorks became a formal product in 1987.

      rest of article [securityfocus.com]

  • by millette (56354) <robin@@@millette...info> on Sunday February 22, 2004 @01:49AM (#8354127) Homepage Journal
    Wow, I didn't expect the rover had 128MiB of RAM, or 256MiB of flash. Funny to think they had to run chkdsk from so far away :)
  • Space Technology (Score:5, Insightful)

    by superpulpsicle (533373) on Sunday February 22, 2004 @01:49AM (#8354131)
    That's the thing that amaze me. Any technology having to do with space seem that much more advanced.

    Here on earth we can't even build cars that require no maintainance and last more than 10 years.
    • by beeplet (735701) <beeplet@gmail.com> on Sunday February 22, 2004 @01:57AM (#8354174) Journal
      Actually any technology making it into space is more likely to be 10 years out of date... Getting anything certified for space is a long process. The technology in space isn't more advanced, just much better documented and well-understood.
      • by kfg (145172) on Sunday February 22, 2004 @02:12AM (#8354248)
        Ten years out of date, but ten years more reliable for the effort.

        Sort of like Debian.

        Cutting edge ain't always what it's cracked up to be.

        KFG
      • Here's an example of the Mars Rover's 10 year old networking technology:

        Ring, Ring, Ring....
        "Welcome to the Mars Rover answering system. For English press 1, Para Espanol prensa 2"
        BEEP

        "You selected English. To leave a message for Spirit press 1. To leave a message for Opportunity press 2"
        BEEP

        "You selected Spirit. Transfering now." CLICK "I'm sorry, Spirit is unavailable at this time. To leave a message press 1. To return to the main menu press 2"
        BEEP

        "Hi this is the Spirit rover. I can't come to the

    • Re:Space Technology (Score:4, Informative)

      by DerekLyons (302214) <fairwater&gmail,com> on Sunday February 22, 2004 @01:51PM (#8356235) Homepage
      That's the thing that amaze me. Any technology having to do with space seem that much more advanced.

      Here on earth we can't even build cars that require no maintainance and last more than 10 years.
      Most of the stuff in space that lasts ten years usually has no moving parts, which is what generates much of the maintenace requirements on your car. Nor does it have parts to get fouled, corroded, or otherwise mucked up by the enviroment of or the operation of the car.

      And frankly, if your car isn't lasting ten years, then you bought junk in the first place. Of the four cars I've owned, not one has had a lifetime of less than ten years. Three of them were already older than that when then they came to me, and none lasted me less than four years. (Other than the one that got re-possesed, but I had that one three years.) But then I invest in regular maintenance, don't leadfoot, etc...
  • by Anonymous Coward on Sunday February 22, 2004 @01:50AM (#8354135)
    I hope they use SSH or something .. who's to say a future mission ..some hax0r doesnt grab control of a space probe and have it send goatse.cx pics back??

    All it takes is a transmitter out in the middle of nowhere africa or some island .. after all the probe communicates using known frequencies. There may be probs picking up the return signal without an expensive antenna i suppose. But then again maybe some hax0r can build one cheaply and or do what captin midnight did ( www.signaltonoise.net/library/captmidn.htm ).

    I wouldnt worry about signal jamming though as that will probably be discovered easily.
    • by mcbridematt (544099) on Sunday February 22, 2004 @01:59AM (#8354184) Homepage Journal
      I don't think they would bother using anything to do with TCP. Anything you do send you will have to wait 9 minutes for. Just imagine the ping times:

      Pinging mars-rover with 32 bytes of data:
      request timed out
      request timed out
      request timed out
      64 bytes from mars-rover: icmp_seq=0 ttl=64 time=32400ms :(

      If it has anything to do with current internet protocols, it would be UDP.
      • by AhBeeDoi (686955) on Sunday February 22, 2004 @04:00AM (#8354542)
        ttl=64
        I realize that Mars is a long way away, but how many routers do you think exist between here and there?
      • by Anonymous Coward on Sunday February 22, 2004 @05:06AM (#8354703)
        UDP would be even worse. Interplanetary transmission is difficult, so some packet loss is likely. Under UDP the packets would just disappear-it's an unreliable protocol. TCP would of course be too inefficient. I'd expect them to use a custom protocol designed for the specific application, since their situation is totally unlike anything you'll face on Earth.
  • by Tablizer (95088) on Sunday February 22, 2004 @01:53AM (#8354148) Homepage Journal
    The Martians are pissed that the repair labor was outsourced to Earth.
  • by prakslash (681585) on Sunday February 22, 2004 @01:56AM (#8354166)
    Unless you are a lay person, I don't understand what the big deal is .

    If it was the hardware that got fried and they miraculously fixed that, I would understand but this was just a software glitch.

    I routinely reboot and reprogram machines in our data-center that is 2000 miles away from me.

    As long as all hardware components are working and there is connectivity to the machine, it doesn't matter whether the machine is a few miles away or a million miles away.

    • by Gizzmonic (412910) on Sunday February 22, 2004 @02:06AM (#8354219) Homepage Journal
      I routinely reboot and reprogram machines in our data-center that is 2000 miles away from me.

      As long as all hardware components are working and there is connectivity to the machine, it doesn't matter whether the machine is a few miles away or a million miles away.


      You are too humble, friend. What you do routinely and without thinking, is nothing less than a miracle of modern science. A miracle that you take part in every day. And because of men like you, we don't have to rely on the abacus anymore. We sent a pentium to the Moon, and soon, Mars will be colonized by G5s. America salutes you, for all the things that you do.....

      Like a rock! I was strong as I could be be!

      Ooooooohh! Like a rock!
    • by dellis78741 (745139) on Sunday February 22, 2004 @02:15AM (#8354262)
      The tricky part here was that the 'hardware connectivity' depended on 'software functionality'. Try maintaining machine a block away if the commnication link requires both ends to point a satellite dish at an orbiting satellite and that pointing relied of software functioning correctly.
    • by FTL (112112) * <slashdot@neil[ ]aser.name ['.fr' in gap]> on Sunday February 22, 2004 @02:19AM (#8354280) Homepage
      I routinely reboot and reprogram machines in our data-center that is 2000 miles away from me.

      As long as all hardware components are working and there is connectivity to the machine, it doesn't matter whether the machine is a few miles away or a million miles away.

      There are some fundamental differences, my friend:

      • If you screw up leaving the computer unbootable, you get local tech support to check the console and fix it. NASA on the other hand doesn't have tech support on Mars.
      • If you hose the server, it means a day's worth of reinstallation. If NASA hoses their rover, they just lost $300,000,000.
      • You can poke around the system and see what's wrong. NASA has a harder time since their lag time is 20 minutes.
      • You can download core dumps, NASA were operating on the low-bandwidth antenna which meant looking at file sizes, time stamps, selected lines, but not file contents.
      • You have your boss breathing down your neck (hoping for success), NASA have the international media breathing down their necks (hoping for a disaster).
      • by jelle (14827) on Sunday February 22, 2004 @04:38AM (#8354625) Homepage
        But at NASA, you have a local replica of the whole system sitting in the lab next door, you're in a team of professionals that if necessary can calculate the most probable results of particular radiation hitting your system under a given angle, or can tell you the power usage and temperature effect of the system components given a particular subroutine, or can dream low-level correct assembly for the platform under study, plus the vendor has a couple of on-line support guys sitting in chairs in the corner of your office waiting for your activation command (which is the word "huh?")...

    • by updog (608318) on Sunday February 22, 2004 @02:21AM (#8354287) Homepage
      There is a big difference between this, and your example of forcing a controlled reboot of your remote machines.

      Spirit was in a constant reboot cycle, and the fact that they could even communicate with it long enough to bypass the problem was an accomplishment (and lucky).

      It would be more similar to your remote data-center machine suddenly going offline and you have no idea why, and you are unable to ssh to it, and you fix it by running through potential scenarios and finding that the problem could have been due to mounting a certain partition, then discovering that there's an exploit in ICMP that allows you to hack to kernel so it doesn't mount that partition.

      • by Matrix9180 (734303) * <matrix9180.gmail@com> on Sunday February 22, 2004 @05:28AM (#8354740)
        Did you RTFA? The rover was rebooting over and over because it was using up all of it's memory... then eventually the batteries were low so it went into a sort of 'safe mode' where only the absolute minimum was loaded, and that's when NASA was able to communicate with it again...

        It was nothing like what you described, just a VERY well designed system (though it would have been somewhat better had the system been able to go straight to "safe mode" after the initial critical error (running out of memory))

        Did the people with mod points RTFA? Score 5 Insightful?

        And no, I'm not new to /. ;)
    • by amRadioHed (463061) on Sunday February 22, 2004 @02:24AM (#8354303)
      Are you forgetting that the latency when communicationg with mars averages around 1200000 ms? I'd say that when you have to wait 20 minutes to see the result of anything you do you're going to have to substantially change your debugging strategy.
      • by cookiepus (154655) on Sunday February 22, 2004 @03:47AM (#8354520) Homepage
        I'd say that when you have to wait 20 minutes to see the result of anything you do you're going to have to substantially change your debugging strategy.

        Please! Back in the day people would write programs on paper, mail them in an envelope to a computing center somewhere, and get results weeks later.

        THAT was pressure not to fuck up.

        • There was also pressure not to drop your stack of punched cards in those days!

          (hint - draw a diagonal line across their top edges so you can get them in order again quickly.)

          Some people seem to no know why "batch" files were so-called, it seems.

          YAW.
    • by afidel (530433) on Sunday February 22, 2004 @02:27AM (#8354315)
      Actually I remember NASA doing a hardware repair from most of the way across the solar system. One of the deep space probes was starting to have a problem sending signals, some bright mind at NASA looked at the circuit diagram and figured out that a single component (resistor, cap, can't remember) was starting to fail, they figured out that there was a way to recondition the part. So they came up with a program that basically intentionally overstressed that component path and the extra energy heated up the part an reconditioned it so that the unit was back to working condition.
    • by NymblZ (754889)
      As long as all hardware components are working and there is connectivity to the machine, it doesn't matter whether the machine is a few miles away or a million miles away.

      That's just it - consider the stress those rovers are enduring or might encounter: subzero tempatures down to -200f, out-of-the-blue (red?) sandstorms, gamma radiation, and who knows what else out there that could suddenly fsck with the systems or scramble internal data ? Your average Dell rack will never have to deal with any of thos
    • by Blaede (266638) on Sunday February 22, 2004 @03:33AM (#8354494)
      Today we salute YOU, Mr. Super Wizard Windows Reinstaller.

      Only YOU can fully appreciate the difficulty of running a format c: command, while swilling a room temperature can of Red Bull.

      "Hey this stuff is hard now!"

      While NASA is too preoccupied with things like farway rovers, you take your vocational tech school fueled arrogance directly to the place where it will make the absolute least possible impact: A Slashdot discussion thread.

      "Loggin' on now!"

      Your unique eye for obviousness allows you to sling turds of obtuseness every which way, and then brag about how you were RIGHT as soon as one of your pronouncements hit true - regardless of how many times you were wrong before.

      "See I told you sooooooo!!"

      And if some idiot rocket scientist has the unmitigated gall to not bow down to your obvious Geniusdom, you unleash your fury down upon him with all the tenacity and mercilessness of a rabid pit bull with a tender buttock locked in its jaws.

      "Total anonymity!"

      So keep clicking away, oh Marauder of the Mousepad. Because when the results you so desire finally come about years from now, you can say it was because YOU demanded it."

      "How come they haven't fired that dumbass head of NASA yet yet?"

      (Bud Light Beer, Anheuser Busch, St. Louis Missouri.)
  • Uh-oh (Score:5, Funny)

    by z0ink (572154) on Sunday February 22, 2004 @01:57AM (#8354170)
    "We recognized early in the planning process that the flash file system had a limited capacity for files."

    Sounds like NASA forgot to empty the rover's recycle bin. =)
    • Re:Uh-oh (Score:3, Funny)

      by LnxAddct (679316)
      I've thought long and hard on this topic and yes on windows it is accurately called the recycle bin because you dont get rid of the junk you put in there, it gets reused in some other part of your system. You put junk in, the junk is modified into other junk and then sent back to create new system dlls. In linux(and I believe macs) it is accurately called the trash can because what we put in there is thrown out for good, we don't have our junk recycled to create more, but different, junk:)
      Regards,
      Steve
      • Re:Uh-oh (Score:3, Funny)

        by brendan_orr (648182)
        Nah, Linux, Mac OS X, *BSD, and other *nix users have /dev/null as a trash can.
        • Re:Uh-oh (Score:3, Funny)

          by AhBeeDoi (686955)
          Nah, Linux, Mac OS X, *BSD, and other *nix users have /dev/null as a trash can.
          Trash can? More like a neutron star, 'cause anything you put in it is totally and absolutely gone.
  • The proper fix... (Score:3, Insightful)

    by Dan East (318230) on Sunday February 22, 2004 @01:57AM (#8354173) Homepage Journal
    ...would have been to have "fixed" the problem before the hardware left earth. This "bug" (or more accurately, known limitation of the filesystem) should have been discovered here on earth if the rover had been properly tested.

    The only real bug was the inability of the system to properly handle running out of file entries (or more specifically, consuming too much RAM as the number of file entries increased). However the software should have never have stressed the filesystem to that degree in the first place.

    Dan East
    • by Chester K (145560) on Sunday February 22, 2004 @02:36AM (#8354342) Homepage
      The only real bug was the inability of the system to properly handle running out of file entries (or more specifically, consuming too much RAM as the number of file entries increased). However the software should have never have stressed the filesystem to that degree in the first place.

      When you can write an embedded operating system that can gracefully and automatically recover from every possible thing that might ever go wrong, perhaps you should send your resume to NASA.
    • Re:The proper fix... (Score:5, Informative)

      by KewlPC (245768) on Sunday February 22, 2004 @03:15AM (#8354449) Homepage Journal
      Score: -1, Didn't Read Article

      The rovers were extensively tested before launch. For example, NASA took about 100000 pictures with the test panoramic cameras under varying conditions to see how they would react. NASA put a test rover on a tilting platform to see how far over the rover tilt before it capsized, to find out at what angle the electric motors could no longer drive the rover up a hill, etc.

      This limitation of the filesystem was known about ahead of time. If you had read the article, you'd have known that. They had a utility to clean out the rover's filesystem, but a storm at the Deep Space Network site that was supposed to transmit it prevented the second half of the utility from being uploaded to the rover. And before you say anything else, the article also mentioned that the people involved had thought of this possibility ahead of time.
  • Hindsight (Score:5, Insightful)

    by FTL (112112) * <slashdot@neil[ ]aser.name ['.fr' in gap]> on Sunday February 22, 2004 @01:57AM (#8354175) Homepage
    The article (I know, I know, this is Slashdot) is really good. It contains everything that is missing from traditional media. The story, the background, technical details, and follow through.

    Granted mainstream media have to keep their coverage dumbed down if Joe Public are going to read it. But what really bugs me is the lack of follow-up. We hear about poorly understood events as they are unfolding, then never heard about them later when they are completely understood.

    A recent example is the gangway between ship and shore at the QM2's drydock. It collapsed killing lots of people, an investigation was launched. Why did it collapse? At the time it wasn't known. I'm sure it's known now, but there's been absolutely no followup.

    This article about the rover is great not so much because of the level of detail but because it reports on an event with the benefit of hindsight.

    • Re:Hindsight (Score:3, Informative)

      by Jeremy Erwin (2054)
      I'm sure there will be at least some mention of the results of the investigation when it is completed and various persons are prosecuted. In the meantime, here's a relatively recent article [yahoo.com] on the investigation into the collapse.
    • Re:Hindsight (Score:5, Interesting)

      by Anonymous Coward on Sunday February 22, 2004 @04:16AM (#8354588)
      I'm a journalism undergrad at a large university. One of the points I brought up with some of our administrators is that the innumeracy and scientific illiteracy of the graduates of our program is appalling. I think this is one reason why many important stories don't get reported accurately or in depth: the writers simply don't understand the story, and don't want to understand the story. They actually feel that math and science are somehow beneath them, and that the average reader doesn't need to be bothered with the facts. So we get vagueness instead of specifics in the articles we read.

      I suggested we allow j-students to substitute math or hard science minors in place of the foreign language requirement. Most graduates of college foreign language programs don't translate at a level any higher than Babelfish. It seems wasteful to force people to spend so much time learning a language that most will never use, when that time could be more productively spent introducing them to the languages of math and science, which they will undoubtedly use in the future. We'd get better reporting that way, and isn't that what going to j-school is all about? Science and technology are too important to our day-to-day lives and governance to be left to illiterates.

  • by Mr2cents (323101) on Sunday February 22, 2004 @02:05AM (#8354213)
    What filesystem is used? Is wear leveling being used? The directory structure is apparently stored in RAM during the day (why else would it use so much RAM?), that is a good thing for reducing wear on the flash system. But what's the number of writes on the flash chips? When will that number be reached?
    • by afidel (530433) on Sunday February 22, 2004 @02:32AM (#8354329)
      Never, the rovers are only going to operate for ~100 days, the number of writes for modern flash ram is 100K cycles minimum, over a million typical. So unless they are really screwing something up that shouldn't be a limitation, also distributing file placement shouldn't be a software function, good CF cards do it in the controller logic.
  • Mod this "redundant" (Score:5, Informative)

    by Penguinshit (591885) on Sunday February 22, 2004 @02:06AM (#8354221) Homepage Journal

    'How do you diagnose an embedded system that has rendered itself unobservable?'

    The way you do this is by having an exact duplicate of the remote system so you can set up a test with conditions as close to those under which the remote system is currently operating. You can then do a series of carefully controlled test solutions to determine the optimum prior to trying it on the "live" system.

    This is the way I set up all my production systems and, barring catastrophic hardware failure (self-immolating disks and a router which just folded when its power supply burped) I've had perfect uptime.

    (well, ok.. there was that one time, late at night, when I typed "reboot" in the wrong window.. but that happens...)

  • Lucky Hack? (Score:5, Insightful)

    by electromaggot (597134) on Sunday February 22, 2004 @02:11AM (#8354243)
    "The outcome strikes me as an extremely Lucky Hack..."

    The outcome does not strike me as a "Lucky Hack." They made the system flexible, that flexibility got them into some trouble, and it's also what got them out of it. Anyone else agree?
    • Yes, see my post (Score:3, Insightful)

      by SuperKendall (25149) *
      I had pretty much the same post - the originator of the story confuses luck with skill, a mistake a find very annoying and committed all too frequently. I'll fully admit when I've been lucky, but I also went recognition for foresight when I've had some! NASA deserves at least that much respect.
  • by dmeranda (120061) on Sunday February 22, 2004 @02:11AM (#8354246) Homepage
    "The irony of it was that the operating system was doing exactly what we'd told it to do," Klemm lamented.

    Yeah, that was HAL's excuse too.

    Seriously, hats off to all the JPL programmers. Proving to the Martians that there is indeed intelligent life on Earth, very intelligent.

  • by Peter McC (24534) on Sunday February 22, 2004 @02:14AM (#8354261) Homepage
    My pet peeve when I'm doing remote troubleshooting is 'ifconfig eth0 down'...oops. At least NASA is smarter than that.

    Peter.
  • Lucky Hack? (Score:5, Insightful)

    by SuperKendall (25149) * on Sunday February 22, 2004 @02:21AM (#8354291)
    Your post is the only thing that strikes me as a "Lucky Hack" here. They included the ability in the design to remotely disable booting from flash and upload new boot images, in what way is that a "hack"? All this is just foresight in design to include as many possible recovery modes as they could.

    Basically, they rebooted from a recovery image (sent via radio) and then proceeded to do low-level fixes on Flash memory and they a chkdisk. If I do something similar via recovery disk or CD, I don't get a lot of people telling me that it was a "Lucky Hack" that I could boot off of CD!!!
  • NASA Rocks! (Score:5, Interesting)

    by blueZhift (652272) on Sunday February 22, 2004 @02:23AM (#8354301) Homepage Journal
    Great article! This is just the sort of thing that has always impressed me about NASA and the JPL. Just when mere mortals might give it up and walk away, they figure out the problem. I can only imagine how wild the party must have been after they fixed Spirit, the scientists and engineers I've worked with in the pass could really put away the booze.

    Seriously though, the key lessons to take away from this are.

    1) Gather all of the clues you can.

    2) Take those clues and build a model.

    With luck and care, the model should get you closer to what may have gone wrong. And in this case it apparently did just that. Now that's geek cool!

    BTW, I know that generally you want to prevent this sort of thing from happening. But in reality most software ships with bugs and launch windows to Mars are non-negotiable.

  • Remote safe mode (Score:4, Interesting)

    by Megane (129182) on Sunday February 22, 2004 @02:27AM (#8354314) Homepage
    The first thing needed to achieve remote maintainability on the order of space probes is some way to access a machine remotely when it's not running the full OS. A KVM switch isn't going to work over long distances. The BIOS needs a way to run over the network. Same for the kernel boot messages. Whether it's through a serial console and SSH server, or through the BIOS running TCP/IP, what we have now isn't enough. A separate console server could also control a power cycle/reset switch circuit.

    There also needs to be a way to load bootstrap code remotely. For instance, having a TCP/IP enabled BIOS be able to run TFTP or some other protocol to load a netboot floppy image. Then you could give it a LILO command instructing it where to find a boot image, preferably one on a server in the same hosting center.

  • whoops (Score:5, Funny)

    by usillyman (755322) on Sunday February 22, 2004 @02:38AM (#8354349)
    Operating System not found. Press any key to continue.
    Damn! Left the floppy in!
  • by AaronStJ (182845) <AaronStJNO@SPAMgmail.com> on Sunday February 22, 2004 @02:38AM (#8354350) Homepage
    What surprises me is that they don't have a 'twin' of the rover's computer system set up on earth. When commands are run on the rover, the same commands could be run on the computer system on earth. Then, if the rover's software, fails (as it did), the software on earth would (theoretically) fail in a similar way, and be MUCH easier to debug. Of course, the systems wouldn't be identical (without building an entire duplicate and expensive rover), and the data gatehred wouldn't be identical, but if the twin was carefully planned and fed dummy data that aproximately mirrored that data the rover was gathering. For example, the twin could be fed dummy pictures about as often as the rover took a real picture.

    From the article "[The] transmission that uploaded the utility was a partial failure: Only one of the utility program's two parts was received successfully. The second part was not received, and so in accordance with the communications protocol it was scheduled for retransmission on sol 19." NASA could have simulated a half failed transfer on the twin copmuter on earth, and then watched carefully using traditional debugging tools to make sure the failed transmission didn't cause a software failure (which it did).

    Again, from the article "The data management team's calculations had not made any provision for leftover directories from a previous load still sitting in the flash file system." However, if they had a twin computer system to watch, they would have seen that the failure occur on earth as it did in space. Debugging a system you can hook a serial debugger to is bound to much easier than debugging a system a million miles away.
    • by Anonymous Coward on Sunday February 22, 2004 @02:51AM (#8354383)
      Uhmm... we DID build a 'twin' of the rover, hardware and all. Give us a bit more credit, will ya? :-P What you may not realize is that exposure to radiation on the surface of Mars, solar wind while in transit and other factors such as thermal expansion / contraction, etc. are slowly degrading the rovers in nondeterministic ways. It is not nearly as simple as 'running the commands in the testbed' at JPL to diagnose any problems which occur.
    • They do have a twin system here, but having one here isn't quite the same as the two on Mars. You can't replicate everything on the two Mars rovers such as the science data files.

      When Spirit was turned around on it's lander, they tested the moves on it's twin here, hence the long delay getting off the lander.

    • Brilliant! (Score:4, Funny)

      by frenchs (42465) on Sunday February 22, 2004 @03:26AM (#8354474) Homepage
      They could have set it up out in my backyard to take pictures of the piles of crap and rocks out there and if they wanted to simulate the solar radiation, they could have my girlfriend give it one of her famous looks... cause those are leathal enough to burn a hole in your soul.

      -SF
  • by Anonymous Coward on Sunday February 22, 2004 @02:41AM (#8354360)
    .. namely, "Do Not Use VxWorks". Use something stable instead. eCos [planetmirror.com] comes to mind. So does everyone's favorite OS these days [kernel.org], which has RTOS support. Having been a frustrated VxWorks user in the past, I'd no more entrust my mission-critical services to it than I would to Microsoft. -- TTK
  • by superyooser (100462) on Sunday February 22, 2004 @03:13AM (#8354439) Homepage Journal
    The operating system is Wind River Systems' Vx-Works version 5.3.1, used with its flash file system extension.

    First wxWindows [slashdot.org], now Vx-works?

  • by nsayer (86181) <nsayer@kfCOWu.com minus herbivore> on Sunday February 22, 2004 @03:17AM (#8354456) Homepage
    Before doing something risky, type this:

    sleep 600 && reboot &

    Now if your risky maneuver makes the ssh session unusable, just wait 5 minutes for the machine to reboot.

    This is great for fiddling with firewalls by remote control... through the firewall. :-)

    Oh... You say you're not using a POSIX-like system? That's not supported. Sorry. :-)
  • by vinit79 (740464) on Sunday February 22, 2004 @03:30AM (#8354489)
    What really surprises me is that NASA did not verify the software. Software verification [google.com] is essentially mathematically proving the software. It is tedious and expensive but we are talking about NASA and the Mars. Infact even beloved MS formally verifies device drivers [microsoft.com] before use ( believe it or not !!) If the original program was correct they wouldnt have to reupload it and the entire problem ...gone.
    • by WayneConrad (312222) <[moc.ingay] [ta] [darnocw]> on Sunday February 22, 2004 @04:50AM (#8354649) Homepage

      Software verification is essentially mathematically proving the software....

      I've been hearing how great formal verification is since I started this gig. Three decades later, it's still not what Yourdon and his buddies thought it would be. When the first computer scientists were budded from mathematics departments, their mathematical discipline allowed them to do wonderful things, some of which we're still catching up with. But it also gave them some disturbing habits, the worst of which is the insistence that formal verification is the best way to write code, and anyone not doing so must be a fool.

      Formal verification is a powerful tool, but as you say, it is expensive and applies to only a limited set of problems. If it were so cheap and so widely applicable, we'd be using it everywhere.

      We've poured decades of funding into formal verification, but the useful tools keep coming from other avenues of research. I think it's time to stop beating the formal verification drum.

  • by PipoDeClown (668468) on Sunday February 22, 2004 @04:13AM (#8354577)
    is because when the batteries got drained the os went into a stable "safe mode" state. If they made a long lasting powersupply this project was doomed(.f) and they never found out what the real problem was.
  • What we can learn: (Score:5, Insightful)

    by sakusha (441986) on Sunday February 22, 2004 @04:14AM (#8354579)
    It appears that we still haven't learned the biggest lesson of all. I still remember back around 1970, there was a big sign on the wall next to the IBM 370s at my university, written on a primitive pen plotter, it said:

    Computers never make mistakes, they do exactly what humans tell them to do. All "computer errors" are human errors.

  • JPL (Score:3, Funny)

    by EachLennyAPenny (731871) on Sunday February 22, 2004 @06:49AM (#8354852) Homepage
    The JPL is a pretty viral license. It forces you to spread their space probes from your planet to all your customer's planets. This is un-solar systematic! What's next? Calling GNUpiter Jupiter instead?
  • Hmmmm (Score:4, Insightful)

    by ziggy_zero (462010) on Sunday February 22, 2004 @07:07AM (#8354873)
    "The irony of it was that the operating system was doing exactly what we'd told it to do"

    Funny, that's how it was explained to me by my computer science teacher my freshman year in high school. He said, "The problem with computers is that they do exactly what we tell them to."
  • by thrill12 (711899) on Sunday February 22, 2004 @07:31AM (#8354890) Journal
    "We discovered a system log in which the problem was documented,"
    Those guys are running a very expensive experiment, are logging it and they have no idea what and where they are logging??
  • by sommerfeld (106049) on Sunday February 22, 2004 @11:58AM (#8355642)

    It's not that hard to pull off off this sort of seemingly amazing remote recovery with pure off-the-shelf tech if you plan for it in advance and are willing to pay a modest premium.

    You need remote serial console access -- ideally including firmware/bios serial console access -- and remote power cycling, controlled by a small embedded system, either in separate units (APC masterswitch, terminal servers) or as part of the system unit (common on Sun gear as "LOM"/"ALOM"/etc.; some of this is also creeping into x86 mobos). All this lets you regain control of the system remotely.

    Then it becomes a matter of hardening the system to let you recover from various other insults. Never let go with both hands: Mirrored disks (protecting against hardware failure) and multiple bootable partitions (protecting against software or human error) can both be used; netbooting is also a nice capability to have when you've got a bunch of servers in the same place.

    Disclaimer: I bet you can do much of the above with other people's gear, but I work for Sun and I know it works for me...

"I have just one word for you, my boy...plastics." - from "The Graduate"

Working...