Debugging The Spirit Rover 390
icebike writes "eeTimes has a story on how the Mars Rover was essentially reprogrammed from millions of miles away. 'How do you diagnose an embedded system that has rendered itself unobservable? That was the riddle a Jet Propulsion Laboratory team had to solve when the Mars rover Spirit capped a successful landing on the Martian surface with a sequence of stunning images and then, darkness.' The outcome strikes me as an extremely Lucky Hack, and the rover could have just as likely been lost forever. Are there lessons here that we can use here on the third rock for recovery of our messed up machines which we manage from afar via ssh?"
Mod this "redundant" (Score:5, Informative)
'How do you diagnose an embedded system that has rendered itself unobservable?'
The way you do this is by having an exact duplicate of the remote system so you can set up a test with conditions as close to those under which the remote system is currently operating. You can then do a series of carefully controlled test solutions to determine the optimum prior to trying it on the "live" system.
This is the way I set up all my production systems and, barring catastrophic hardware failure (self-immolating disks and a router which just folded when its power supply burped) I've had perfect uptime.
(well, ok.. there was that one time, late at night, when I typed "reboot" in the wrong window.. but that happens...)
not really... (Score:4, Informative)
Another factor in this is the safety of the flash ram. It is rad-hardened and built with tons of extra error correction which again, requires years of testing and special design considerations. And is extremely expensive.
Re:Space Technology (Score:2, Informative)
Re:Hindsight (Score:3, Informative)
Re:NASA should have simulated... (Score:4, Informative)
Re:Could an earthbout 'twin' computer help? (Score:4, Informative)
Re:Could an earthbout 'twin' computer help? (Score:3, Informative)
When Spirit was turned around on it's lander, they tested the moves on it's twin here, hence the long delay getting off the lander.
Ran out of INODES. No really. (Score:5, Informative)
To me, if this were a Unix-like system, it sounds like they ran out of inodes [webopedia.com]. Running out of inodes is very different than running out of disk space.
If you think runing out of disk space can be hard to trouble shoot, try running out of inodes.
Re:The proper fix... (Score:5, Informative)
The rovers were extensively tested before launch. For example, NASA took about 100000 pictures with the test panoramic cameras under varying conditions to see how they would react. NASA put a test rover on a tilting platform to see how far over the rover tilt before it capsized, to find out at what angle the electric motors could no longer drive the rover up a hill, etc.
This limitation of the filesystem was known about ahead of time. If you had read the article, you'd have known that. They had a utility to clean out the rover's filesystem, but a storm at the Deep Space Network site that was supposed to transmit it prevented the second half of the utility from being uploaded to the rover. And before you say anything else, the article also mentioned that the people involved had thought of this possibility ahead of time.
Verifying the software !!! (Score:4, Informative)
Re:only 120 megs ram? (Score:5, Informative)
The point is to use well known and well tested hardware. The whole point of Mars Pathfinder was to develop a system whose design could be re-used for other Mars landers and rovers.
Lastly, what exactly are you going to do with greater flash capacity? The point of having any flash memory on the rovers at all is not for long term storage, but rather just to hold onto data until it can be transmitted to Earth, after which it gets deleted.
Despite what some idiot posted a few posts up, they did NOT run out of room on the flash drive. Rather, the problem is more akin to running out of i-nodes. Mounting the flash filesystem, reading all its metadata and whatnot, took up more RAM than was allocated for it, due to the high number of files it had to deal with (most of which were accumulated on the way to Mars, and were going to be deleted).
The only reason nasa got it back to work (Score:3, Informative)
Re:WindRiver's fault (Score:3, Informative)
Re:Ran out of flash disk space. No, really. (Score:3, Informative)
After a few days on Mars, they were starting to fill up the flash, so they planned to go ahead and delete the old launch OS image, its directories and files. This is a complicated process so they uploaded a special program to do it on Sol 15. And apparently they informed the rest of the team that the memory would be free and available after that point, so the rest of the team made plans to start filling it up with pictures.
However, the upload on sol 15 failed, and was rescheduled for sol 19. Now, here's the big mistake (which the article glosses over): They forgot to tell the rest of the team that all that memory wasn't going to be freed up as planned, not for a few more days. So instead, Spirit is moving around now, taking lots of pictures, storing them in flash, and all the people involved with that think they have plenty of room. Little do they know that they are running out of flash space. Finally, the morning of Sol 19, shortly before the memory cleaning program was going to be sent down, it happened. The flash memory was exhausted. This triggered a sequence of events which put the craft into a failure loop.
The big problem here, then, was the failure on the part of the group which was supposed to clean out the launch OS image to tell the rest of the team that it wasn't going to happen as scheduled, so the memory wasn't going to be available. It wasn't really Murphy's Law, but rather a failure to communicate among the team. This is an institutional problem which will hopefully be fixed.
Re:Space Technology (Score:3, Informative)
Cadillacs I don't really know too well, but I know a BMW doesn't need a whole lot of repairs. Most german cars are VERY well built. Much better than japanese cars too. And what good's a car that'll last you forever if you don't like the piece of shit in the first place. I just bought a new car (My mustang's in NY, my sister drives it now, my grandmother didn't like the idea of me driving across country in a 1990 Mustang with 300+ rwhp, on such long straight roads, top speeds 145 btw), I could have gone with a VERY cheap Honda Civic, it would probably last me most of my adult life but why would I want such a piece...? I bought a fully loaded 2004 Nissan Sentra SE-R SpecV, it's a quick car, with low insurance and great looks. I wouldn't have bought anything less, I didn't look into TCO at all, it didn't really matter to me. I don't want a car that'll last me forever if I don't like it. And most people let the dealer pick out the car they want, they don't really realize it but they don't care, the average person wants to get from point A to point B, and the salesman is gonna try to sell them a car that costs a lot of money, not caring about the life of the car.
Re:rebooting on mars... (Score:2, Informative)
2. That's not what "begs the question" means. http://skepdic.com/begging.html [skepdic.com]
3. Based on 1 and 2, it is proved by example that you=monkey puppet.
Re:What the article doesn't say (Score:1, Informative)
Re:Ran out of flash disk space. No, really. (Score:4, Informative)
It was the inability to build the RAM-based directory structure of the files in the Flash memory.
Why couldn't they build the directory structure? They had too many files, the size of the files doesn't matter here, only the number of files.
In other words, they ran out of RAM, not Flash.
Exercise left for the readers: Why can a Unix file system that is out of inodes have much less than 100% disk usage and still not be able to create a file?
Re:One reasonable anology (Score:5, Informative)
Have a hardware watchdog. If the machine is lost or confused, it reboots itself.
Have it come up in a known state, fire off a few broadcast packets to the sysadmins, and run sshd but basically nothing else. Stay there for a minute or so.
If nobody's tried to log in and halt the boot process, carry on booting. With luck the problem was transient. Worst case the problem still exists, you reboot, and the admins get another chance to log in.
From the description of how they got Spirit back, it looks like this is exactly how it was set up.
Who'da thunk it!!
Re:lots of mem of an embedded system (Score:3, Informative)
A highly respected embedded OS.
YAW.
Parent should be modded down (Score:3, Informative)
The problem was discovered after launch. The first few fixes made the problem worse by stressing the filesystem even further.
It doesn't matter that they were trying to fix the problem. THAT WAS NOT MY POINT. The problem should have been identified and fixed before the craft was launched.
Yes, they may have taken "around" 100000 pictures. Does that mean they sequentially stored every picture in an actual rover file system? I get the impression they were only testing the cameras or the capture software, not the holistic system.
Did they first simulate filling the filesystem with files generated during the actual trip to mars? Apparently not, because the system would have failed if they had actually put the rover software through a launch to end of mission simulation here on earth when the software was developed.
Dan East
Re:They didn't just randomly delete stuff (Score:2, Informative)
"...Separately, about 230 Mbytes are used to implement a flash file system..."
Re:Oh, sure... (Score:4, Informative)
There is a low gain omni-directional antenna that can be used as backup. Infact I think they use it most of the time for commands and just use the high-gain for data transfer back to Earth. Which makes sense, they never need to send large amounts of data to the rover.
No lightning has ever been detected on Mars. Tho it's not impossible, it is very very unlikely. No proper observations of the night side of Mars has been done tho, so they may just be missing it.
And Opportunity did fall into a hole
Re:Oh, sure... (Score:4, Informative)
Actually though, it's not too bad an analogy. While Earth based servers aren't absolutely unreachable like SPirit, they are often remote, and there are expenses associated with visiting them in person.
Various schemes now exist to help deal with that. Many boards have a small management processor (bmc, server management board, IPMI, whatever) that is used for remote diagnostics and reconfiguration when the main board won't even boot.
Meanwhile, LinuxBIOS supports two complete BIOS images. One 'old reliable' that once working is never changed, and one that can be upgraded freely. Coupled with a watchdog card or timer, it's decently managable in the field. That work is continuing.
Meanwhile, IBM is pushing the 'blue button' that forces a software reload from an image partition.
In that sense, the problem is strongly analogous. Most of us will not, however, encounter the exact problem that Spirit had, though some embedded device developers just might.
Re:Great trick for ssh administration (Score:2, Informative)
Ergo, if everything works and you don't want to reboot anymore, you just do a little % followed by a little ctrl-C and it's all good.
Incidentally, sleep 600 will make the machine sleep for 10 minutes, not 5 minutes, as the OP said :)
you, too, can have this capability on earth... (Score:4, Informative)
It's not that hard to pull off off this sort of seemingly amazing remote recovery with pure off-the-shelf tech if you plan for it in advance and are willing to pay a modest premium.
You need remote serial console access -- ideally including firmware/bios serial console access -- and remote power cycling, controlled by a small embedded system, either in separate units (APC masterswitch, terminal servers) or as part of the system unit (common on Sun gear as "LOM"/"ALOM"/etc.; some of this is also creeping into x86 mobos). All this lets you regain control of the system remotely.
Then it becomes a matter of hardening the system to let you recover from various other insults. Never let go with both hands: Mirrored disks (protecting against hardware failure) and multiple bootable partitions (protecting against software or human error) can both be used; netbooting is also a nice capability to have when you've got a bunch of servers in the same place.
Disclaimer: I bet you can do much of the above with other people's gear, but I work for Sun and I know it works for me...
Re:Space Technology (Score:4, Informative)
And frankly, if your car isn't lasting ten years, then you bought junk in the first place. Of the four cars I've owned, not one has had a lifetime of less than ten years. Three of them were already older than that when then they came to me, and none lasted me less than four years. (Other than the one that got re-possesed, but I had that one three years.) But then I invest in regular maintenance, don't leadfoot, etc...
Re:What's the big deal?? (Score:3, Informative)
Re:do they use SSH ? (Score:3, Informative)
Because very strong forward error correction coding is used, the link tends to be "brittle"; as long as you stay just under the maximum allowable data rate, it will work perfectly. So a lot of work goes into making those accurate link predictions.
But data can still be lost if the signal-to-noise ratio takes an unexpected dip. The most likely cause is rain at the earth station site, as the weather is not as easily predicted and water is a strong absorber of X-band radio energy. Most of the DSN sites are in deserts for just this reason. But even if data is lost, it can be retransmitted later as it is stored on the rover until explicitly deleted.
Re:What's the big deal?? (Score:3, Informative)
The failed components never recovered, but JPL was able to work around it. They constructed an elaborate thermal model of the spacecraft to predict the precise temperature (and therefore the operating frequency) of the command receiver. Everything but the kitchen sink went into this model: the effect of attitude on solar heating, the self-generated heat from the electronics, the effect of turning various instruments on or off, the time lags due to structural heat capacities, everything. And it has worked fine ever since.
JPL doesn't get nearly the credit they deserve for their track record in rescuing missions from seemingly fatal failures like these. There's still a pervasive public myth (sustained by the human space flight side of NASA) that only humans in space can fix things when they break. But they seriously overestimate the astronauts' abilities, and they greatly underestimate what a bunch of really smart people can often do from the ground.
Re:What we can learn: (Score:3, Informative)
You still haven't learned the lesson. Those are not errors or mistakes, they are malfunctions. A properly designed computer system can easily detect malfunctions
Guess you'd better get over to NASA and set up a series of lectures so that you can impart your vast expertise and wisdom.
but that same system will happily execute any human-designed code containing massive errors.
Interesting that you point out the code as being human-designed. Who designed the hardware? God?
You're just the kind of computer geek I abhor, always looking for excuses instead of solutions to your own mistakes.
And you're just the kind of self-assured idiot who amuses me endlessly with your clueless but oh-so-confident assertions.
In the real world, hardware defects do exist, some designed into the hardware, others induced by external effects or damage. Software errors are certainly far more common, but that's mostly just because there's vastly more software.
Even without the effects of space travel, hardware contains flaws and, indeed, much of the job of low-level software is to work around those flaws. It's not uncommon for a significant percentage of the code in a device driver to be dedicated to working around various hardware defects.
Anyone who's spent considerable time working around custom and embedded computing hardware knows that defects often turn out to be *both* hardware and software-based. Insignificant hardware bugs interact with insignificant software bugs to produce major problems. Hardware defects aren't limited to those environments, either. Spend a little time searching the LKML archives for "ACPI" and reading what you find, or even just look through the Linux kernel configuration help and see how many configuration options you find that implement softare hacks to work around problems with particular pieces of hardware.
When you factor in the rather unique and harsh operating environment of this hardware and software, and consider the amount and depth of testing that certainly went into the development process, it's not in the least bit unusual that the programmers should be surprised that the flaw was purely a software error. If I'd been in those engineers' shoes, I also would have expected something far more complex. I'm sure they went into it, quite reasonably, assuming that some hardware component had failed and that they were going to have to implement a software workaround.
I'm sure the prevailing sentiment when they finally discovered the actual nature of the problem was "Hallelujah! This is something we can fix!", not "Uh, oh, I can't blame this on anyone else." That's certainly how I would have felt, anyway.