Spirit 'Will Be Perfect Again'

G. Holst writes "NASA technicians are preparing to wipe Spirit's flash memory clean of science and engineering files that have stymied its software. The fix, likely to be made Friday, could completely restore Spirit. "I think it will be perfect again," says the Mission Manager. Chalk this one up for earth!" There are numerous stories about Spirit and Mars: one describes being careful with rm -rf. Reader Tablizer sends in an interesting site: "I discovered Bill Momsen's website where he describes his experiences working on the first successful photographic mission to another planet: Mariner IV to Mars."

flash...

[by Anonymous Coward]

I'm reminded of the unforgettable Queen song:


Flash, a-ha, saviour of the universe

Flash, a-ha, he'll save everyone of us.....

Re:flash...

[KingDaveRa (620784)]

Flash, a-ha, its full of files Flash, a-ha, run rm -rf

perfect again?

[knitting fool (542573)]

".. I think it will be perfect again.." meaning that it was perfect the first time...?

Re:perfect again?

[by Anonymous Coward]

Even perfecter. The next rover mission will be even more perfecter. Maybe even the perfectest!

Re:perfect again?

[sbeitzel (33479)]

Reminds me of a saying my friends and I had back during undergrad CS classes: "It was perfect, so I fixed it." This explains a lot about software development.

Ctrl-Alt-Del sent to mars...

[ivanmarsh (634711)]

Woo-hoo!

Glad to hear Spirit will be feeling herself again.

Re:Ctrl-Alt-Del sent to mars...

[AndroidCat (229562)]

Herself? It fell asleep with its probe stuck out. Now it's going to wake up with some memories missing. That sounds like a guy activity to me.

Like that joke..

[ph43thon (619990)]

A chemist, an engineer and a computer scientist are passing through a vast desert in a car when suddenly the car breaks down.

"Goddamnit! There must have been some sudden increase of enthalpy in the cylinder!" the chemist yells, gnashing his teeth, banging on the steering wheel.

"Maybe the fan belt broke or the battery is dead or the wheels came off.." the engineer mumbles.

After thinking a while the computer scientist shrieks in a shrill, frantic voice:

"Let's just try getting out of the car and getting back in!!!@!"

p

Another variant

[ptbarnett (159784)]

A manager, an engineer, and a programmer are in a car, on their way to a meeting. As they descend down a hill, the brakes fail. The driver manages to slow the car down and stop it on the side of the road. All three get out and ponder what to do next.

The manager says: "First, we have to appoint a committee to investigate the problem and recommend a solution. Then, we must write a project plan, and review the specifications before we can start design and implementation. I estimate it will require about 3 months."

The engineer says: "I have some tools in my briefcase. I can rebuild the master brake cylinder in an hour or so, and we'll be on our way."

The programmer says: "No, no, no! First, we have to push the car up to the top of the hill and see if the brakes fail again!"

My question

[aliens (90441)]

For all you kernel and OS heads out there. Was this primarily due to shitty software being run on the rover?

I mean could VxWorks be responsible for not being able to function with the Flash RAM filled?

Re:My question

[by Anonymous Coward]

VxWorks in my experience is terrible at memory management, and when you get close to the "edge" it becomes almost useless. Not just with Flash memory either. Even when managing a very large disk system I always try to keep at least 20% free.

Re:My question

[Mr2cents (323101)]

Even if the memory handling is shitty, I wonder how it could have caused so much havoc.. How could it have caused spirit to go into the reset loop? It seems like some bad error handling code was also in play here (just guessing, the details aren't public to my knowledge..).
Another thing that surprised me is that if the flash had been broken, all data had to be uploaded before the rover went to sleep.. every modern PC can continue to refresh it's DRAM while sleeping. Why can't spirit? Maybe a feature to consider on future missions?

Re:My question

[Ralph Wiggam (22354)]

Someone here with VXworks experience explained this a few days ago. To be safe, the system reboots when a memory allocation request fails. It sounds like Windows tech support, step 1: reboot computer. The workaround is to prevent those requests from failing.

-B

Re:My question

[confused one (671304)]

It's not so much that VxWorks reboots when a memory allocation request fails. It that the memory allocation request will cause the kernel to crash & later a watchdog timer will interrupt the processor & force it to reset.

VxWorks memory, embedded protection

[devphil (51341)]

Released versions of VxWorks do not have protected memory. (The development version does.) So nothing is there to prevent overwrites by concurrent tasks, etc.

Those of you in the audience experienced in embedded systems know that this makes sense for embedded hardwar -- VxWorks or not -- for three main reasons:

1. Stuff running in such environments is damn near bug-free. It's not like, say, Mozilla, or even the Linux kernel, or even /bin/ls. These things get tested rigourously, not as an afterthought deligated to the junior programmer.

2. In systems which are allowed to fail once in a while, reboots are fast. There's no hard drive to spin up, no filesystem to fsck, etc. It can just go *click* and humans won't typically see an interruption in [whatever it was the doohickey was doing].

3. There's usually no point in memory protection. If the propulsion system walks off the end of a garbage pointer, mission's over. No real use in keeping the guidance system going; it's already on a ballistic uncontrollable arc. If some critical part of the super-smart pacemaker fails (see #1), there's no victory in digging the device out of the corpse and saying, see, this other critical part wasn't affected, thanks to the memory protection! In those cases, memory protection just increases the cost and size of a device, without helping anything.

Protected memory is good for systems which do more than one thing, and/or have parts which can die without killing the whole device (e.g., a desktop computer). And as I said above, some embedded OSes are added such protection for customers who want to adapt their technology to more general-purpose tasks.

Re:VxWorks memory, embedded protection

[nathanh (1214)]

Stuff running in such environments is damn near bug-free. It's not like, say, Mozilla, or even the Linux kernel, or even /bin/ls. These things get tested rigourously, not as an afterthought deligated to the junior programmer.

That's false reasoning.

1. No practical software is bug-free.

2. Testing is never complete.

3. People make mistakes, even during testing.

4. Spirit broke down.

It makes sense, when building a robust system, to do rigorous testing AND have the memory protection.

VxWorks obviously has a brilliant team of brainwashers^Wsalesmen because they've convinced you that you don't need a feature they don't offer. Perfect!

Re:My question

[PineGreen (446635)]

Yes, actually it seems to be a filesystem bug... I mean, a reasonably stable filesystem - every OS has this, I am really surprised they messed this up! I wouldn't mind if it was an obscure kernel race condition or something, but filesystem!!!

Re:My question

[Docrates (148350)]

well, the way I understand it is this: remember how in old DOS (and other OS's) you had to set the number of files open?

files=30

well, that basically told the OS how many files it was going to have to handle at any given time.

Well, in the case of Spirit, it's not that they were short on flash or RAM, it's that the portion of RAM used to handle the files in flash when the flash filesystem is mounted grew unexpectedly for some reason (kinda like the frames in conventional memory you used to access extended memory in DOS). They think the problem was that this portion of RAM used to handle Flash files was not big enough for the amount of files they had in the flash (including files from 6-7 months in transit and a couple of days on the ground in mars).

Soooo, a quick (ok, maybe not so quick) rewrite of the routines in the OS for this flash-files-handling-RAM-portion should do the trick.

Bottom line, it WAS a bug that could only surface with thousands of files in flash, which is something they didn't try on the ground.

Re:My question

[crawling_chaos (23007)]

Bottom line, it WAS a bug that could only surface with thousands of files in flash, which is something they didn't try on the ground.

Which is a reminder to always test the boundary conditions, no matter how ridiculous they may seem. If it is possible to have that many files, then the regression test scripts should generate that many files during testing.

At least it's fixable.

Re:My question

[techiemac (118313)]

Ok ok ok... chill out everyone...
VXWorks is not that bad (I use it on almost a daily basis). Every single OS has its problems. Before we all go and start calling VXWorks or Spirits software a crappy piece of code, you have to understand what goes into writing space qualified software.
This is not some thing you hack together over the weekend. In fact something you wrote for a space system over the weekend would be tested over a period of months and possibly even years depending on the criticality of the code. We're talking life critical system testing here. That means all paths for you code heads out there.
That said, even when you hit rubber to the road, there are always unexpected situations. Something that you didn't anticipate, a bug that made its way through under circumstance x. Hands up for everyone here who has written a complex bug free system right out of the gates. Anywone who just lifted their hand does not understand what a complex system is or a bug. Though stuff that flies tends to be pretty darn close to bug free.
We are dealing with many complex unknowns when we land something on another planet.
VXWorks is actually very popular with the space program. It's not perfect but neither is Linux (though someday it will be right ;) ). In fact the whole system that they are using on the rover has flown quite a few times (VXWorks running on rad hardened PowerPCs with a VME bus for it's backbone).
Trust me, the software running on the rover is not crappy. In fact, the fact they can bring it back to life like they did says a lot.

The attempt failed

[AtariAmarok (451306)]

(AP) "Attemps to wipe the flash memory clean on the Spirit rover failed today, when it was found out that someone flipped those tiny plastic switches to "protect" on the SD memory cards that are serving the unit.

A press conference is expected tomorrow to announce sending someone to Mars to set the SD cards to allow erasing."

Courageous engineers!

[EulerX07 (314098)]

These guys are about to wipe the memory of a robot on another planet and they're confident and casual. Just flashing the bios of my motherboard in my computer room causes me anguish and fills me with terror...

Re:Courageous engineers!

[TaKiNiTeZ (747064)]

Some people claim that it also helps keeping their spirit up.

Re:Courageous engineers!

[damien_kane (519267)]

Depending on what it is they are consuming and in what quantities, their spirits may be coming up quicker than they expect...

Re:Courageous engineers!

[javatips (66293)]

it's true... I had the opportunity to keep keep my spirit up for so long that I have memory flashes all the time.

This is all a conspiracy.

[Scoria (264473)]

NASA technicians are preparing to wipe Spirit's flash memory clean of science and engineering files that have stymied its software.

Obviously, this is an attempt to suppress the discovery of alien life on Mars. After a "severe communications fault," NASA is destroying the "scientific" data collected by Spirit. Coincidence? I think not.

I postulate that Echelon (yes, that Echelon) intercepted a message being transmitted by the alien race. Yes, our government subsequently disabled the probe to prevent successful reception!

It's a frickin' lobotomy, man!

[burgburgburg (574866)]

Spirit won't go along with the agenda of "the man", so they're taking away it's individuality, man. They just want to make it their robot, taking their photographs and doing their experiments. But will they invite it over for dinner? Will they let it date their daughter? Did they even give it a round-trip ticket? No, no and no, man.

Early Spring Cleaning?

[Wiser87 (742455)]

I'm surprised that they had kept the files that were to be used only during the cruise stage.(source: www.spaceflightnow.com )
Anyone here know why they bothered to keep the files? Wouldn't they want as much space as possible for the scientific data?

Re:Early Spring Cleaning?

[Jarnis (266190)]

The same reason why your hard drive is cluttered with old unused files.

Why delete, when you still have room on the flash and you *just* might need that file later...

Of course they then found out that their filesystem handler borks out way before the flash is actually filled up, and that almost bought the whole show to an end... Software QA testing failure in my books, but they seem to be recovering from the fumble pretty well...

Repeat?

[sabrex15 (746201)]

One has to wonder, is opportunity going to forego the same problems as spirit?.. As they are "identical" robots.. have steps been put in place to prevent the 2nd robot from "getting full".. I should certainly hope that we dont want this to happen again, as they might not be as lucky to regain it.

Re:Repeat?

[WhiteWolf666 (145211)]

As I understand it, the first thing they did once they got Opportunity on the ground was to clear out all the spaceflight 'cruise' data.

I imagine that someone is keeping an eye on it.

I'm disappointed

[by Anonymous Coward]

How come we're not awash with "Spirit was willing, but flash was weak" jokes?

Re:I'm disappointed

[Scoria (264473)]

The Spirit was willing, but the flash was weak.

Dave?

[by Anonymous Coward]

"Dave, stop ... Stop, will you? Stop, Dave ... Will you stop, Dave ... Stop, Dave. I'm afraid ... I'm afraid ... I'm afraid, Dave ... Dave ... my mind is going ... I can feel it ... I can feel it ... My mind is going ... There is no question about it. I can feel it ... I can feel it ... I can feel it ... I'm a ... fraid ... "

What's happening?

[thung226 (648591)]

They're saying mad-scientist-esque things like "I think it will be perfect again" and calling rocks "Cake."

They've officially lost it.

Mars Rover

[Fenis-Wolf (239374)]

The information coming in about the Rover's the last few days has been fascinating. I never really appreciated the kind of tech that went into these things. It really makes you sit back and think about how very far our species has come in the last 150 years. I mean Jules Verne was only begining to imagine landing on the moon while riding around England in a steam locomotive, now, 150 years later, we routinely launch things into orbit around the Earth, and land radio controlled machines on other planets to roam around.
This is truly a wonderful age to live in.

On the radio...

[Bimo_Dude (178966)]

I heard the headline that "Opportunity has plunged into the atmosphere of Mars," and I couldn't help adding to this in my head, "as well as Spirit, Motivation, Job Prospects, and Hope."

What really happend

[LupusUF (512364)]

Their ISP received a subpoena from the RIAA. NASA is now wiping the memory in hopes that lawyers will not find kazaa and the 1,000 mp3s that are on Spirit.

"Perfect again"

[DrCode (95839)]

Yes, that's what I always say when I fix a bug.

Pretty much OT but an interesting question

[nizo (81281)]

This is kind of a continuation of an earlier post in a different thread, but I wonder who owns these probes? When we eventually send colonists to Mars, are they free to pick apart these things, lug them back to base as decorations, etc. I am guessing the "possession is 9/10ths of the law" fits pretty well here, even though I would bet NASA would throw a hissy fit if some other country took one of the rovers back to base to use as a boot scraper.

Directory Names

[TaKiNiTeZ (747064)]

"... The we have done file deletes on the spacecraft before, so we've shown that does work. The file directories have all different names and you can convince yourself that you are actually deleting the right thing."

I am rather glad they gave all the directories different names. If they had managed to do otherwise, I would not go so far any more as to call the thing they have a "filesystem".

Might even be a future news: "NASA integrates first non-deterministic filesystem into space probe 'Hope'".

Going around JPL

[angst_ridden_hipster (23104)]

My old man, who works at JPL, says that the current phrase going around campus is:

"Spirit is willing, but the Flash is weak."

And people wonder why NASA's budget keeps getting cut.

Sounds like my first IBM clone

[Slick_Snake (693760)]

It was used and a little warn out. The data was intact on the hard drive, but the motor just couldn't get in going on a cold boot. I would have to spin the motor by hand and throw the switch. Ah, the good old days.

Spirit just needs a good jump start. Anyone got some really long jumper cables?

Security?

[DoctorHibbert (610548)]

Other than the huge costs of transmissions equipment, does anyone know what kind of security they use to prevent hackers from doing this (like for instance some mischievous Russian space program scientist)?

Re:rm -rf?!

[stevesliva (648202)]

Actually, Spirit's problems began when one of the NASA engineers created a file named "-rf" in his home directory.

Re:Any theories on what caused the corruption?

[Wiser87 (742455)]

Actually, it looks like it wasn't a case with the flash memory being corrupted, but rather it having to many files...

"We also yesterday completed a scan of the flash memory. This provided us with some important diagnostic information. We are now able to tell that when we mount the flash memory, it does in fact take a lot of the system RAM in the process. In fact, more system RAM than is available. So that's helping confirm the theory we had that the reason the restarts were hanging up was because we were running out of memory when we are trying to mount the flash memory.

Re:Any theories on what caused the corruption?

[AKAImBatman (238306)]

Apparently it was simply too many files and the FS ran out of inodes. Remember that they're constrained to a 256MB file system. It wouldn't surprise me if they used an 8 bit or 16 bit number for the inode count. (Ah, the joys of Vx(Doesn't)Works.)

On another note, does anyone know exactly what they're deleting here? While I understand that they need to get this mission underway, is there a chance they could lose valuable mission or navigational information?

Re:Any theories on what caused the corruption?

[confused one (671304)]

They're deleting all the telemetry and science data Spirit's taken since launch. The OS is in the EEPROMS. With one exception, they can repeat all of the measurements & photos that will be lost. The exception: As one of the orbiters happened to fly directly overhead it took some atmospheric measurements; and, simultaneously Spirit performed the same measurement from the ground -- This would have given them a full thickness measurement of what was going on in the atmosphere at that moment.

Re:What Filesystem?

[morcheeba (260908)]

I wrote some demo fibre channel block-device drivers [mc.com] for vxWorks a few years ago, and I found that VxWorks's FAT FS was buggy. The bug only affected one flavor (I don't remember if it was FAT12, 16, or 32), but it was clearly reproducible and clearly an OS fault. It was a corner case and we found some way around it (like avoiding an writes with a length of 1MB).

Here's the usual rant you see here on slashdot, and it's true: since it was closed source, we couldn't verify that we'd caught all the bad cases, and we couldn't submit the fix to back to WindRiver.

Re:Backup ROM?

[confused one (671304)]

Ummmm, there is. The OS is in the EEPROM. That's how they recovered it: reboot from EEPROM with the Flash disk turned off.