Forgot your password?
typodupeerror
Space Science

Spirit 'Will Be Perfect Again' 331

Posted by michael
from the we-get-signal-main-screen-turn-on dept.
G. Holst writes "NASA technicians are preparing to wipe Spirit's flash memory clean of science and engineering files that have stymied its software. The fix, likely to be made Friday, could completely restore Spirit. "I think it will be perfect again," says the Mission Manager. Chalk this one up for earth!" There are numerous stories about Spirit and Mars: one describes being careful with rm -rf. Reader Tablizer sends in an interesting site: "I discovered Bill Momsen's website where he describes his experiences working on the first successful photographic mission to another planet: Mariner IV to Mars."
This discussion has been archived. No new comments can be posted.

Spirit 'Will Be Perfect Again'

Comments Filter:
  • flash... (Score:4, Funny)

    by Anonymous Coward on Friday January 30, 2004 @02:40PM (#8137176)
    I'm reminded of the unforgettable Queen song:


    Flash, a-ha, saviour of the universe

    Flash, a-ha, he'll save everyone of us.....

  • by knitting fool (542573) on Friday January 30, 2004 @02:42PM (#8137203)
    ".. I think it will be perfect again.." meaning that it was perfect the first time...?
  • by ivanmarsh (634711) on Friday January 30, 2004 @02:42PM (#8137207)
    Woo-hoo!

    Glad to hear Spirit will be feeling herself again.
    • by AndroidCat (229562) on Friday January 30, 2004 @02:46PM (#8137270) Homepage
      Herself? It fell asleep with its probe stuck out. Now it's going to wake up with some memories missing. That sounds like a guy activity to me.
    • by ph43thon (619990) on Friday January 30, 2004 @03:09PM (#8137522) Journal
      A chemist, an engineer and a computer scientist are passing through a vast desert in a car when suddenly the car breaks down.

      "Goddamnit! There must have been some sudden increase of enthalpy in the cylinder!" the chemist yells, gnashing his teeth, banging on the steering wheel.

      "Maybe the fan belt broke or the battery is dead or the wheels came off.." the engineer mumbles.

      After thinking a while the computer scientist shrieks in a shrill, frantic voice:

      "Let's just try getting out of the car and getting back in!!!@!"

      p

      • by ptbarnett (159784) on Friday January 30, 2004 @03:43PM (#8137895)
        A manager, an engineer, and a programmer are in a car, on their way to a meeting. As they descend down a hill, the brakes fail. The driver manages to slow the car down and stop it on the side of the road. All three get out and ponder what to do next.

        The manager says: "First, we have to appoint a committee to investigate the problem and recommend a solution. Then, we must write a project plan, and review the specifications before we can start design and implementation. I estimate it will require about 3 months."

        The engineer says: "I have some tools in my briefcase. I can rebuild the master brake cylinder in an hour or so, and we'll be on our way."

        The programmer says: "No, no, no! First, we have to push the car up to the top of the hill and see if the brakes fail again!"

  • My question (Score:5, Interesting)

    by aliens (90441) on Friday January 30, 2004 @02:43PM (#8137215) Homepage Journal
    For all you kernel and OS heads out there. Was this primarily due to shitty software being run on the rover?

    I mean could VxWorks be responsible for not being able to function with the Flash RAM filled?
    • I think so.

      Check out this recent Slashdot article [slashdot.org].
    • Re:My question (Score:5, Informative)

      by Anonymous Coward on Friday January 30, 2004 @02:49PM (#8137302)
      VxWorks in my experience is terrible at memory management, and when you get close to the "edge" it becomes almost useless. Not just with Flash memory either. Even when managing a very large disk system I always try to keep at least 20% free.
      • Re:My question (Score:5, Insightful)

        by Mr2cents (323101) on Friday January 30, 2004 @03:07PM (#8137498)
        Even if the memory handling is shitty, I wonder how it could have caused so much havoc.. How could it have caused spirit to go into the reset loop? It seems like some bad error handling code was also in play here (just guessing, the details aren't public to my knowledge..).
        Another thing that surprised me is that if the flash had been broken, all data had to be uploaded before the rover went to sleep.. every modern PC can continue to refresh it's DRAM while sleeping. Why can't spirit? Maybe a feature to consider on future missions?
        • Re:My question (Score:5, Informative)

          by Ralph Wiggam (22354) * on Friday January 30, 2004 @03:21PM (#8137653) Homepage
          Someone here with VXworks experience explained this a few days ago. To be safe, the system reboots when a memory allocation request fails. It sounds like Windows tech support, step 1: reboot computer. The workaround is to prevent those requests from failing.

          -B
          • Re:My question (Score:5, Informative)

            by confused one (671304) on Friday January 30, 2004 @03:35PM (#8137807)
            It's not so much that VxWorks reboots when a memory allocation request fails. It that the memory allocation request will cause the kernel to crash & later a watchdog timer will interrupt the processor & force it to reset.
          • by devphil (51341) on Friday January 30, 2004 @03:59PM (#8138026) Homepage


            Released versions of VxWorks do not have protected memory. (The development version does.) So nothing is there to prevent overwrites by concurrent tasks, etc.

            Those of you in the audience experienced in embedded systems know that this makes sense for embedded hardwar -- VxWorks or not -- for three main reasons:

            1. Stuff running in such environments is damn near bug-free. It's not like, say, Mozilla, or even the Linux kernel, or even /bin/ls. These things get tested rigourously, not as an afterthought deligated to the junior programmer.

            2. In systems which are allowed to fail once in a while, reboots are fast. There's no hard drive to spin up, no filesystem to fsck, etc. It can just go *click* and humans won't typically see an interruption in [whatever it was the doohickey was doing].

            3. There's usually no point in memory protection. If the propulsion system walks off the end of a garbage pointer, mission's over. No real use in keeping the guidance system going; it's already on a ballistic uncontrollable arc. If some critical part of the super-smart pacemaker fails (see #1), there's no victory in digging the device out of the corpse and saying, see, this other critical part wasn't affected, thanks to the memory protection! In those cases, memory protection just increases the cost and size of a device, without helping anything.

            Protected memory is good for systems which do more than one thing, and/or have parts which can die without killing the whole device (e.g., a desktop computer). And as I said above, some embedded OSes are added such protection for customers who want to adapt their technology to more general-purpose tasks.

            • There's usually no point in memory protection. If the propulsion system walks off the end of a garbage pointer, mission's over. No real use in keeping the guidance system going; it's already on a ballistic uncontrollable arc. If some critical part of the super-smart pacemaker fails (see #1), there's no victory in digging the device out of the corpse and saying, see, this other critical part wasn't affected, thanks to the memory protection! In those cases, memory protection just increases the cost and size o
            • by nathanh (1214) on Friday January 30, 2004 @06:43PM (#8139536) Homepage
              Stuff running in such environments is damn near bug-free. It's not like, say, Mozilla, or even the Linux kernel, or even /bin/ls. These things get tested rigourously, not as an afterthought deligated to the junior programmer.

              That's false reasoning.

              1. No practical software is bug-free.

              2. Testing is never complete.

              3. People make mistakes, even during testing.

              4. Spirit broke down.

              It makes sense, when building a robust system, to do rigorous testing AND have the memory protection.

              VxWorks obviously has a brilliant team of brainwashers^Wsalesmen because they've convinced you that you don't need a feature they don't offer. Perfect!

              • It makes sense, when building a robust system, to do rigorous testing AND have the memory protection.

                Absolutely. While building it.

                VxWorks obviously has a brilliant team of brainwashers^Wsalesmen because they've convinced you that you don't need a feature they don't offer. Perfect!

                I forgot, this is slashdot, where VxWorks is the eternal enemy, and second-guessing actual rocket scientists is the national sport.

                IIRC, memory protection was removed from the early versions by popular request, b

        • Re:My question (Score:3, Insightful)

          by techiemac (118313)
          To answer your question, there was probably a watchdog timer that caused it to go into a reset loop.
          Yes modern PCs have all of these wiz bang features but let me ask you this... would you want to be on an airplane where it's fly by wire system was controlled with your PC? No probably not.
          Systems that fly and are life critical (yes there is no one on it, but space systems are held to that standard) cannot have a bunch of wiz bang features on board. The more you add, the more potential for failures. So you tr
    • Re:My question (Score:5, Insightful)

      by PineGreen (446635) on Friday January 30, 2004 @03:00PM (#8137434) Homepage
      Yes, actually it seems to be a filesystem bug... I mean, a reasonably stable filesystem - every OS has this, I am really surprised they messed this up! I wouldn't mind if it was an obscure kernel race condition or something, but filesystem!!!
    • The thing has a pretty decent bandwidth to earth; why not just dd the filesystem to earth, wipe it, and start afresh?
      • oh sure, and risk a mistake? something like "dd if=/dev/zero of=/dev/earth"? a giant laser beam would emerge from spirit and rewrite earths surface causing chaos and destruction.

    • Re:My question (Score:5, Interesting)

      by Docrates (148350) on Friday January 30, 2004 @03:19PM (#8137637) Homepage
      well, the way I understand it is this: remember how in old DOS (and other OS's) you had to set the number of files open?

      files=30

      well, that basically told the OS how many files it was going to have to handle at any given time.

      Well, in the case of Spirit, it's not that they were short on flash or RAM, it's that the portion of RAM used to handle the files in flash when the flash filesystem is mounted grew unexpectedly for some reason (kinda like the frames in conventional memory you used to access extended memory in DOS). They think the problem was that this portion of RAM used to handle Flash files was not big enough for the amount of files they had in the flash (including files from 6-7 months in transit and a couple of days on the ground in mars).

      Soooo, a quick (ok, maybe not so quick) rewrite of the routines in the OS for this flash-files-handling-RAM-portion should do the trick.

      Bottom line, it WAS a bug that could only surface with thousands of files in flash, which is something they didn't try on the ground.
      • Re:My question (Score:5, Insightful)

        by crawling_chaos (23007) on Friday January 30, 2004 @03:42PM (#8137884) Homepage
        Bottom line, it WAS a bug that could only surface with thousands of files in flash, which is something they didn't try on the ground.

        Which is a reminder to always test the boundary conditions, no matter how ridiculous they may seem. If it is possible to have that many files, then the regression test scripts should generate that many files during testing.

        At least it's fixable.

        • Re:My question (Score:3, Informative)

          by DerekLyons (302214)

          Docrates:

          Bottom line, it WAS a bug that could only surface with thousands of files in flash, which is something they didn't try on the ground.

          crawling_chaos:

          Which is a reminder to always test the boundary conditions, no matter how ridiculous they may seem. If it is possible to have that many files, then the regression test scripts should generate that many files during testing.

          The problem here isn't so much one of boundary conditions, but of the subtle differences between simulators and reality.

          None

    • Re:My question (Score:5, Insightful)

      by techiemac (118313) <techiemac@yaTEAhoo.com minus caffeine> on Friday January 30, 2004 @03:24PM (#8137692)
      Ok ok ok... chill out everyone...
      VXWorks is not that bad (I use it on almost a daily basis). Every single OS has its problems. Before we all go and start calling VXWorks or Spirits software a crappy piece of code, you have to understand what goes into writing space qualified software.
      This is not some thing you hack together over the weekend. In fact something you wrote for a space system over the weekend would be tested over a period of months and possibly even years depending on the criticality of the code. We're talking life critical system testing here. That means all paths for you code heads out there.
      That said, even when you hit rubber to the road, there are always unexpected situations. Something that you didn't anticipate, a bug that made its way through under circumstance x. Hands up for everyone here who has written a complex bug free system right out of the gates. Anywone who just lifted their hand does not understand what a complex system is or a bug. Though stuff that flies tends to be pretty darn close to bug free.
      We are dealing with many complex unknowns when we land something on another planet.
      VXWorks is actually very popular with the space program. It's not perfect but neither is Linux (though someday it will be right ;) ). In fact the whole system that they are using on the rover has flown quite a few times (VXWorks running on rad hardened PowerPCs with a VME bus for it's backbone).
      Trust me, the software running on the rover is not crappy. In fact, the fact they can bring it back to life like they did says a lot.
  • by AtariAmarok (451306) on Friday January 30, 2004 @02:45PM (#8137239)
    (AP) "Attemps to wipe the flash memory clean on the Spirit rover failed today, when it was found out that someone flipped those tiny plastic switches to "protect" on the SD memory cards that are serving the unit.

    A press conference is expected tomorrow to announce sending someone to Mars to set the SD cards to allow erasing."
  • by doc_traig (453913) on Friday January 30, 2004 @02:45PM (#8137243) Homepage Journal

    "It's the only way to be sure."

  • by EulerX07 (314098) on Friday January 30, 2004 @02:45PM (#8137248)
    These guys are about to wipe the memory of a robot on another planet and they're confident and casual. Just flashing the bios of my motherboard in my computer room causes me anguish and fills me with terror...
  • by Scoria (264473) <slashmail@@@initialized...org> on Friday January 30, 2004 @02:46PM (#8137260) Homepage
    NASA technicians are preparing to wipe Spirit's flash memory clean of science and engineering files that have stymied its software.

    Obviously, this is an attempt to suppress the discovery of alien life on Mars. After a "severe communications fault," NASA is destroying the "scientific" data collected by Spirit. Coincidence? I think not.

    I postulate that Echelon (yes, that Echelon) intercepted a message being transmitted by the alien race. Yes, our government subsequently disabled the probe to prevent successful reception!
  • Normally I would be against just getting out the restore disc and starting over... it usually just causes the problem to come back later.

    In this case I think it's a good idea. I'd hate to have to go to Mars to make a housecall on a sick puter

  • scary (Score:2, Funny)

    by dan2550 (663103)
    that sounds a little too similar to what i did with my fancy TI graphing calculator last week...
  • by Wiser87 (742455) on Friday January 30, 2004 @02:47PM (#8137273) Homepage
    I'm surprised that they had kept the files that were to be used only during the cruise stage.(source: www.spaceflightnow.com )
    Anyone here know why they bothered to keep the files? Wouldn't they want as much space as possible for the scientific data?
    • by Jarnis (266190) on Friday January 30, 2004 @03:09PM (#8137525)
      The same reason why your hard drive is cluttered with old unused files.

      Why delete, when you still have room on the flash and you *just* might need that file later...

      Of course they then found out that their filesystem handler borks out way before the flash is actually filled up, and that almost bought the whole show to an end... Software QA testing failure in my books, but they seem to be recovering from the fumble pretty well...
  • Repeat? (Score:5, Insightful)

    by sabrex15 (746201) on Friday January 30, 2004 @02:48PM (#8137287)
    One has to wonder, is opportunity going to forego the same problems as spirit?.. As they are "identical" robots.. have steps been put in place to prevent the 2nd robot from "getting full".. I should certainly hope that we dont want this to happen again, as they might not be as lucky to regain it.
  • by Anonymous Coward on Friday January 30, 2004 @02:48PM (#8137293)
    How come we're not awash with "Spirit was willing, but flash was weak" jokes?
  • "We're going to have to blow ROM".
  • Dave? (Score:5, Funny)

    by Anonymous Coward on Friday January 30, 2004 @02:49PM (#8137304)
    "Dave, stop ... Stop, will you? Stop, Dave ... Will you stop, Dave ... Stop, Dave. I'm afraid ... I'm afraid ... I'm afraid, Dave ... Dave ... my mind is going ... I can feel it ... I can feel it ... My mind is going ... There is no question about it. I can feel it ... I can feel it ... I can feel it ... I'm a ... fraid ... "
  • by thung226 (648591) on Friday January 30, 2004 @02:49PM (#8137306)
    They're saying mad-scientist-esque things like "I think it will be perfect again" and calling rocks "Cake."

    They've officially lost it.
  • For 400 million USD, you better KNOW it will be perfect again before you run mkfs on it.
  • by bxbaser (252102) on Friday January 30, 2004 @02:55PM (#8137370)
    Should have deleted all the porn from the flash memory before it launched.
  • Mars Rover (Score:5, Interesting)

    by Fenis-Wolf (239374) <jbudde@a2teTIGERch.us minus cat> on Friday January 30, 2004 @02:56PM (#8137380) Homepage
    The information coming in about the Rover's the last few days has been fascinating. I never really appreciated the kind of tech that went into these things. It really makes you sit back and think about how very far our species has come in the last 150 years. I mean Jules Verne was only begining to imagine landing on the moon while riding around England in a steam locomotive, now, 150 years later, we routinely launch things into orbit around the Earth, and land radio controlled machines on other planets to roam around.
    This is truly a wonderful age to live in.
  • rm -rf !? (Score:2, Funny)

    by stratjakt (596332)
    there's yer problem.

    Everyone knows that hardware support in any unix sucks monkey dongs. Hell, these cheap bastards probably installed lunix based on all those 10 year old HOWTOs floating around the web.

  • by Bimo_Dude (178966) <{bimoslash} {at} {theness.org}> on Friday January 30, 2004 @03:04PM (#8137457) Homepage Journal
    I heard the headline that "Opportunity has plunged into the atmosphere of Mars," and I couldn't help adding to this in my head, "as well as Spirit, Motivation, Job Prospects, and Hope."
  • by LupusUF (512364) on Friday January 30, 2004 @03:05PM (#8137482)
    Their ISP received a subpoena from the RIAA. NASA is now wiping the memory in hopes that lawyers will not find kazaa and the 1,000 mp3s that are on Spirit.
  • Backup ROM? (Score:2, Insightful)

    by xTown (94562)
    I'm neither a rocket scientist nor a computer scientist, so maybe this is a dumb question, but how come there's not some sort of ROM somewhere in the rover itself that contains a backup of the system in its initial state? Obviously, you'd only use it in a worst-case scenario, but you could restore it and then there'd at least be something and they could reapply all the patches one by one.
  • We can rebuild it. We have the technology.
  • by DrCode (95839) on Friday January 30, 2004 @03:16PM (#8137605)
    Yes, that's what I always say when I fix a bug.
  • Easy as Pie (Score:3, Funny)

    by Anonymous Coward on Friday January 30, 2004 @03:21PM (#8137655)

    To restart your Mars Rover, simply insert your NASA Emergency Restore Disc into the CD drive bay located at the side of the Rover. If Autoplay is enabled, the reinstallation software will start automatically. To start the reinstallation process manually, please see the Service Manual included with your Mars Rover OEM package...

  • by nizo (81281) on Friday January 30, 2004 @03:23PM (#8137680) Homepage Journal
    This is kind of a continuation of an earlier post in a different thread, but I wonder who owns these probes? When we eventually send colonists to Mars, are they free to pick apart these things, lug them back to base as decorations, etc. I am guessing the "possession is 9/10ths of the law" fits pretty well here, even though I would bet NASA would throw a hissy fit if some other country took one of the rovers back to base to use as a boot scraper.
    • This is kind of a continuation of an earlier post in a different thread, but I wonder who owns these probes? When we eventually send colonists to Mars, are they free to pick apart these things, lug them back to base as decorations, etc. I am guessing the "possession is 9/10ths of the law" fits pretty well here, even though I would bet NASA would throw a hissy fit if some other country took one of the rovers back to base to use as a boot scraper.

      I don't know what the general answer to this question is, but
  • by TaKiNiTeZ (747064) on Friday January 30, 2004 @03:26PM (#8137712)
    "... The we have done file deletes on the spacecraft before, so we've shown that does work. The file directories have all different names and you can convince yourself that you are actually deleting the right thing."

    I am rather glad they gave all the directories different names. If they had managed to do otherwise, I would not go so far any more as to call the thing they have a "filesystem".

    Might even be a future news: "NASA integrates first non-deterministic filesystem into space probe 'Hope'".
  • by angst_ridden_hipster (23104) on Friday January 30, 2004 @03:27PM (#8137723) Homepage Journal
    My old man, who works at JPL, says that the current phrase going around campus is:

    "Spirit is willing, but the Flash is weak."

    And people wonder why NASA's budget keeps getting cut.

  • by Slick_Snake (693760) on Friday January 30, 2004 @03:32PM (#8137776) Journal
    It was used and a little warn out. The data was intact on the hard drive, but the motor just couldn't get in going on a cold boot. I would have to spin the motor by hand and throw the switch. Ah, the good old days.

    Spirit just needs a good jump start. Anyone got some really long jumper cables?

  • by IversenX (713302) on Friday January 30, 2004 @04:00PM (#8138037) Homepage
    ...and I have this terrible pain in all the diodes down my left side!

    Flash update you say? I wont enjoy it.
  • by geekpuppySEA (724733) on Friday January 30, 2004 @04:01PM (#8138043) Journal
    It's porn. Someone wanted to be the first guy to get Venus to Mars.
  • Security? (Score:5, Interesting)

    by DoctorHibbert (610548) on Friday January 30, 2004 @04:18PM (#8138182)
    Other than the huge costs of transmissions equipment, does anyone know what kind of security they use to prevent hackers from doing this (like for instance some mischievous Russian space program scientist)?
  • by mnmn (145599) on Saturday January 31, 2004 @12:04AM (#8141849) Homepage
    I would put two 250GB harddisk in the orbiter and make the rovers upload all data they can to the orbiter on each pass, and delete those files automatically. The slower transmission to Earth could then proceed from the disks.

    Each rover uses 256MB flash and so does my 5 megapixel camera. I know for a fact that I can saturate that space fast in a photography frenzy, so I carry a laptop in the car with charger to transfer everything to it if I'll need more pictures.

    Altho the two rovers have been a staggering success on Mars, I am surprised at two overlooks:

    (1) Keeping track of file size and free space.

    (2) What happens if the space is full.

    Even Linux on a measly ARM720T does a much better job.

Programmers do it bit by bit.

Working...