What's Inside the Mars Rovers 458
Captain Zion writes "Space.com has a story about the hardware and software of Mars Rovers Spirit and Opportunity. Basically, they're radiation-shielded, 20MHz PowerPC machines wirh 128Mb RAM and 256Mb of flash memory, running VxWorks. I wonder if I could make a nice firewall with one of these for my home network..."
What's the bus speed on that thing? (Score:4, Funny)
Re:What's the bus speed on that thing? (Score:5, Insightful)
Does a 20mhz processor really need 128mb of ram? I mean, with a bus speed that low, you can probably put the data to flash ROM just as fast. What are the chances of you using all 128mb of ram?
I imagine they can use all the storage they can, since there's no hard drive. So, the RAM acts as a cache for everything that is transmitted (which is a lot, actually). The Flash is used for more permanant software, like OS, commands, other files, ect. I'm amazed they can do it all with as little as they have.
Re:What's the bus speed on that thing? (Score:3, Informative)
Disclaimer: Even faster flash devices may exist, but I don't know
Re:What's the bus speed on that thing? (Score:3, Insightful)
Re:What's the bus speed on that thing? (Score:2)
Re:What's the bus speed on that thing? (Score:5, Informative)
A processor of any speed doesn't need RAM of any size.
The application you want to run needs both processing power and memory. How much of each? Depends on the application.
Re:What's the bus speed on that thing? (Score:4, Interesting)
Re:What's the bus speed on that thing? (Score:4, Insightful)
Re:What's the bus speed on that thing? (Score:4, Interesting)
Re:What's the bus speed on that thing? (Score:2)
Re:What's the bus speed on that thing? (Score:3, Informative)
Re:What's the bus speed on that thing? (Score:5, Funny)
Re:What's the bus speed on that thing? (Score:5, Funny)
Ask Bill Gates.
memory overflow caused Spirit shutdown? (Score:3, Informative)
The lockup happend just as they were going to drill into the rock they've been sitting in front of for nine days. Perhaps there was drill issue too. When the rover memory crashed, it tried to reboot its computer at least a hundred times.
Re:memory overflow caused Spirit shutdown? (Score:3, Funny)
Yeah - everything was going fine until the rover contacted Microsoft for its weekly update and choked on all the security patches.
Re:What's the bus speed on that thing? (Score:4, Insightful)
Radiation hardness (Score:5, Interesting)
Re:Radiation hardness (Score:2)
There have been some incriminating statements along the lines of never having tested the FS for long periods, having to delete extra data from the flash, etc.
Re:Radiation hardness (Score:5, Interesting)
A quote:Raises some interesting questions about software reliability, I think. Did nobody think about running out of disk space?
Re:Radiation hardness (Score:4, Interesting)
Re:Radiation hardness (Score:5, Interesting)
As someone with first-hand experience with VxWorks let me say that VxWorks' memory handling code sucks. Their malloc implementation has got to be the worst one ever designed. It fragments horribly and when fragmented has unusable performance. A malloc call can take many milliseconds when memory gets fragmented. Our box used to crash due to the fragmentation. I replaced Wind River's code with DLMalloc and that fixed the memory issue. We went from many tens of thousands of fragments to only a few dozen. Our reliability significantly increased as well after dumping Wind Rivers brain-dead malloc code. BTW, glibc uses a variant of Doug Lea's malloc code, so it's been widely tested.
Furthermore, in VxWorks there is no way to identify what process malloced a block of memory. There is no memory protection either (think DOS). If a task has a memory leak, VxWorks does not provide any method of tracking down the culprit. I had to add that support to the DL Malloc code I ported so we could find memory leaks and general memory usage by task.
Since malloc is critical to the working of VxWorks, if you run out of memory you are basically completely dead. Often the only way to recover is to wait for the hardware watchdog timer to kick in and reboot the system.
If a task dies in VxWorks with a bad pointer there is no way to recover other than reboot. The OS will not clean up after a task (i.e. free memory, close files, release semaphores).
As far as flash goes, VxWorks supports the FAT16 file system. As you know, FAT16 sucks. It only supports a limited number of files in the root directory. It's relatively easy to corrupt, and when corrupt it tends to corrupt itself even worse. There is no wear leveling support. If the FAT table or root directory gets corrupted, you're screwed. VxWorks is even worse since there arn't tools to fix a corrupt file system.
VxWorks is not a scalable OS. The OS gets slower as the number of tasks increases. Realtime support sucks. Although it has support for things like priority inheritance to prevent priority inversion, the best guaranteed realtime latency is half the system tick rate (the tick rate is usually 10ms).
Also remember that unlike open source operating systems, the source code to VxWorks is not available unless you pay some major $$$. Without the source you're basically working blind.
VxWorks is an old RTOS, and its age is definitely showing. It is not a robust OS.
As far as turning VxWorks into a firewall, you'll need to write all your own code. The VxWorks TCP/IP stack is an archaic vulnerable version of the BSD stack. TCP sequence number guessing is trivial. There is no built-in support for firewall support, NAT, or anything else. I have heard many many complaints about the VxWorks networking code. Although the box I'm working on is a router and broadband remote access server, we don't use the VxWorks TCP/Ip stack much. I am sure the VxWorks stack is vulnerable to many of the current DOS attacks as well.
Some of you might say why not use VxWorks AE, which adds memory protection. Think hacked-in memory protection far worse than Windows 95. It's slow, very buggy, and poorly supported. We tried AE and after many months we dumped it and went back to the non-AE version. Very few companies actually went with AE.
Today there are many other choices for an RTOS that are better than VxWorks. For our next major project we're looking at both QNX and TimeSys Linux. QNX is a true microkernel design with the core being around 70K. Every driver is protected and can be restarted if it crashes. I think you can even buy a medical grade version of QNX. TimeSys Linux is also pretty cool, with excellent real-time support and all the advantages of Linux. For something like the Mars rover, QNX would be better due to the limited amount of memory and greater robustness.
Wi
Re:Radiation hardness (Score:3, Informative)
consider the use of malloc()/free(). In an RTOS it does not much sense to dynamically allocate and free memory at run time. Same thing applied to other sytem resources, like tasks/processes. This takes quite a bit of time from a Real Time point of view to create new tasks, semaphores, ob
Re:Radiation hardness (Score:5, Informative)
I know we're not the only ones to have been burned by Wind River's malloc. I know several major companies that also had to replace Wind River's code.
As far as being able to dynamically replace code, VxWorks isn't alone in that. Numerous other RTOSes out there can do the same thing, including QNX. QNX even supports the concept of a hot standby process to take over if the main process dies.
To give you an idea about how Wind River's malloc works, they keep a sorted linked list of fragments from the smallest to the largest. When you try and allocate a block, it walks the linked list until it finds a block large enough. Likewise, when you free a block it checks if it can coalesc the block with a neighboring block. It then goes through the linked list looking for a slot to insert the free block.
Yes, VxWorks may have been around since the 80's, but that's part of the problem too and it is showing its age. In the 80s embedded processors typically did not have MMUs. Now MMUs are quite common in the more powerful embedded processors.
You say you can't have low latency and memory protection? QNX proves that you can. It is low latency and *very* robust. If your driver dies, no problem, restart it. Timesys Linux also has a very low latency, although not as low as QNX. Timesys also has an interesting feature where you can guarantee CPU and networking resources. I can schedule a task to be guaranteed 5.8ms of execution every 8.3ms and it will guarantee that that task will get the CPU time allotted to it with the desired resolution. This is without increasing the system tick rate (usually 10ms). Timesys can also schedule a task to be higher priority than an interrupt. I'm not as familiar with QNXs scheduler, but it's also quite flexible from what I've heard.
As far as FAT, it is not a robust filesystem. It never has been. If the FAT gets corrupted or a directory entry gets corrupted it's difficult to recover. Other than possibly having 2 copies of the FAT cluster table, any corruption can be difficult to repair. If the FAT table gets corrupted, which table is corrupt and which is not? If a directory entry gets corrupted, it can be impossible to fix. For flash memory, unless you are using a device with special wear-leveling, FAT is about the worst choice since any file write that changes the size of a file requires a write to the directory entry and possibly the FAT table. If the table gets corrupted and you don't run a repair operation (which often ends up leaving orphaned files as lost clusters), the file system can happily corrupt itself to death. Why do you think every time DOS/Windows9x/ME crashed it had to repair the disk with scandisk? FAT is a poorly designed file system that was originally designed for 160K floppies and scales poorly. FAT32 is an improvement, but it's still not very robust. For flash, something like Linux's journalling flash file system 2 (JFFS2) [redhat.com]. More information on VxWorks file system support can be found here [xs4all.nl].
Basic VxWorks information can be found http://www.slac.stanford.edu/exp/glast/flight/docs /VxWorks_2.2/vxworks/guide/ [stanford.edu].
Re:Radiation hardness (Score:3, Insightful)
A) Would never have been chosen as the OS of choice
B) Will never BE chosen to replace VxWorks
Why? It's a great company, but it's based in Ottawa (that's Canada for you Yankees), and NA$A Bucks do not flow over the border.
Re:Radiation hardness (Score:3, Informative)
QNX has a lot of reliability features built in that are not present in VxWorks, Linux, or RTEMS. It can take periodic snapshots of a task and recover if a task dies.
Als
Re:Radiation hardness (Score:4, Informative)
Swapped out? To where? You're being a bit recursive here. The operation failing is the initialization and reading of the flash. You can't swap to flash while you're initializing it.
However, I doubt they implemented virtual memory on the thing. It's far too much overhead in the OS to actually implement paging, etc. They were probably just very, very careful about the amount of memory in use at any one time, as with most embedded systems. Someone, however, didn't check that in a massively fragmented flash filesystem, the directory read wouldn't take up the entire RAM. Oops.
Re:Radiation hardness (Score:4, Insightful)
Re:Radiation hardness (Score:4, Interesting)
What's inside the rover? (Score:5, Funny)
driving the rover (Score:2, Offtopic)
Wait a second... (Score:5, Insightful)
Comment removed (Score:4, Informative)
Re:Wait a second... (Score:3, Informative)
danheskett: That's an excellent point. A lot of people are thinking instruction = 1 cycle. The real world is that it's not unusual for an instruction to take 2, 4, 10, or even 100 cycles. The reality of the matter is that instructions c
Re:Wait a second... (Score:5, Insightful)
Modern superscalar (pipelined) processors have a lot more MIPS than megahertz.
Re:Wait a second... (Score:3, Informative)
Re:Wait a second... (Score:2)
Re:Wait a second... (Score:3, Funny)
Re:Wait a second... (Score:5, Informative)
Go to
http://www.iews.na.baesystems.com/space/rad60
and click on the rover picture to get a PDF brochure, which gives the 33MHz/35MIPS figure.
Rootbear
RAD6000s seem closest to PPC601s (Score:4, Informative)
By all other standards however, they seem to be closely related in time and technology to the 601, which powered Powermac 6100, 7100, 8100, 9150, 7200, 8200 and 7500 PPCs, as well as I think, one of IBM's thinkpads.
Those 601s are a very nice chip, and quite underestimated at what they can do at low clock speeds. If the RAD6000 is anywhere similar, I can understand why it was picked.
Re:RAD6000s seem closest to PPC601s (Score:3, Insightful)
I doubt it, because it probably had little to do with performance. I read an article a while back (tried to Google it now, couldn't find it) that said NASA is generally about ten years behind consumer PC's in terms of the CPU speeds they send into space. These things have to go through so much testing for reliability, etc. that it ofte
Self-warming (Score:5, Funny)
Just leave off the heatsinks and fans, and everything should be fine.
Re:Self-warming (Score:5, Interesting)
If obsessed environmentalists don't like NASA sending up probes with any radioactive material ('it might blow up, ohh..'), then how did this little tidbit get by them? Do they consider it non-radioactive? If they're only concerned by radioactive propulsion systems, then I think they're a bunch of hypocrites. Radioactivitiy is radioactivity whether it's propulsion or heating.
If they don't mind it, then let's send up those dune buggies with RTG and 18-inch wheels and cover a lot more of Mars.
-Cyc
Re:Self-warming (Score:5, Informative)
Radioactivity is NOT radioactivity when you are considering things like this. Saying the people who don't want nuclear powered rockets should hate this as well or else they are hypocrites is tantamount to saying that the people who don't like oil spills should bitch about how some motor oil ALWAYS stays in the plastic container it is shipped in. Not quite the same problem. Afterall, these things aren't much more radioactive than a Coleman lantern wick or a smoke detector element...
Re:Self-warming (Score:3, Insightful)
I don't know, but I'd guess that for rovers of this size and weight, it might be easier to use solar panels than to figure out how to dump all of the waste heat that an RTG would produce. These devices are highly inefficient in terms of converting thermal energy to electrical energy, so they are probably spewing kilowatts of thermal energy at all times, and there's no way to shut them off
Re:Self-warming (Score:5, Informative)
Radioisotope heaters use much less material, as they only need enough heat to keep the warm electronics box above -40F or so. From the Environmental Impact Statement in the Federal Register ([wais.access.gpo.gov][DOCID:fr10de02-54]):
"Each rover would employ two [calibration] instruments that use small quantities of cobalt-57 (not exceeding 350 millicuries) and curium-244 (not exceeding 50 millicuries) as instrument sources. Each rover would have up to 11 RHUs that use plutonium dioxide to provide heat to the electronics and batteries on board the rover. The radioisotope inventory of 11 RHUs would total approximately 365 curies of plutonium."
Nothing you'd like to swallow, but still, much smaller than a radioisotope power unit.
Re:Self-warming (Score:3, Interesting)
Re:Self-warming (Score:3, Interesting)
Re:Sigh (Score:3, Interesting)
*Groan* Uranium is COMMON in asteroids. My engines would use no more than a few tens of pounds of uranium. Worst case, we'll say it uses a few hundred pounds. How is the particulate matter being exhausted by my engines anywhere near the TONS of uranium contained in many asteroids? In fact, most Uranium on Earth is from the tons that burn up in the atmosphere every year.
As for poin
Re:Self-warming (Score:5, Funny)
If you've got money to burn, sure (Score:4, Funny)
Radiation Shielding (Score:4, Interesting)
---
Re:Radiation Shielding (Score:5, Funny)
geeky stuff.. offensive stuff! [wabshirts.com]
Re:Radiation Shielding (Score:5, Informative)
Re:Radiation Shielding (Score:2, Informative)
They have extra shielding on the outside, and the electronics on the inside are designed to disipate sudden charges created by radiation hits.
Re:Radiation Shielding (Score:3, Informative)
Re:Radiation Shielding (Score:2)
They used to use sapphire... (Score:5, Informative)
...as the substrate of the chip, rather than a silicon wafer, so the chip was a "sapphire" chip rather than a silicon chip (although doped silicon could then be used to form transistors, as could Gallium Arsenide or Germanium, through the regular lithographic process).
This is the classic "Silicon On Insulator." IBM has a process of embedding a layer of glass beneath the surface of a standard silicon wafer, allowing SOI using silicon substrates. This and their work with copper set them apart from the other large silicon transisitor foundries (TSMC, Intel, etc.).
The processors on the rovers are probably SOI, but I don't know which process is used.
Redundency Check? (Score:3, Insightful)
Your Average plane have a triple backup system, I spoke to some engineer and he said preflight checks are usally just making sure two of the systems are still working
you'd think they could at least send up some more hardware with these little critters. The extra weight would pan out, when things go bad...case in point see what they are dealing with now :)
Re:Redundency Check? (Score:4, Insightful)
Ofcourse NASA did implement a measure of redundancy by sending two rovers instead of just one.
Re:Redundency Check? (Score:5, Informative)
The online GPC's each carry out the same set of calculations (potentially each uses code designed to do the same thing, but written by different programmers), and they compare each others results. If any single GPC is considered to be too far wrong, the offline GPC's submit their answers. The three GPC's that are in closest agreement then become the new online GPC's, and the remaining two go offline. The GPC's can reboot themselves if they are too far out of whack, if they fail in one of the "results elections", and of course when they are told to do so by the crew.
Also, whenever a GPC is sent offline by one of the others, a specific caution indicator (and potentially the master caution indicator and klaxon) is activated, and the relevant error codes are shown on one of the forward CRT's. The error codes, along with other information such as the currently running program and the current mission phase, determine the crew's actions. Actions can be as simple as disabling the master caution klaxon for the current alert, all the way to hand-checking certain results and manual GPC restarts.
This is all from memory (from about 5 years back), so some of this may have changed recently, particularly on Atlantis with the "glass cockpit" upgrade that happened 18 months or so ago, but the general gist should be about right (and I'm sure I'll soon know if it isn't!!)
Same problem three times (Score:3, Insightful)
A nice Firewall? (Score:5, Funny)
You could, but the latency would be a bitch.
Units, units, units!!! (Score:3, Informative)
Mb = Megabits
MB = Megabytes.
The article writes out megabytes, so MB should be used, not Mb!
Should it have some redundancy? (Score:3, Interesting)
Now, while having two rovers is a form of redundancy, wouldn't it be wise to have some redundancy on each individual rover? I understand that there are concerns like weight and budget, but wouldn't some redundancy be a good form or risk management?
Ouch (Score:3, Insightful)
The bold emphasis is mine but that is a big Ouch for Microsoft.
RAD6000 microprocessors are radiation-hardened versions of the PowerPC chips that powered Macintosh computers in the early 1990s
Shouldn't Apple be using this in their commercials somehow to further boost their reliability. I am sure the PR market can put it in a way that non-geeks watching tv can relate, right?
Re:Ouch (Score:4, Informative)
But try that on an OS for a desktop system, and your email program just may blow up your paint program... (remember Windows 3.1's stability? Make that 10x worse)... You can't use VxWorks for the desktop as Windows is used today... it needs a lot of protection... The ease of upgrading is due to the lack of protection...
Flying VxWorks to Mars (Score:5, Informative)
Flying VxWorks to Mars [colorado.edu]
Who else ... (Score:2)
...read the URL to VxWorks as WinDriver.com instead of WindRiver.com?
Internet conneciton (Score:5, Funny)
Spirit Rover: Staying up longer and harder
Rus
Not 20Mhz (Score:2, Informative)
Something strange... (Score:2, Funny)
http://idisk.mac.com/charlesholt/public/DuckDod
Beagle-2 (Score:3, Interesting)
Save HST! Sell Sojourner knock-offs (Score:5, Interesting)
The profits from Slashdot alone could extend the life of HST or launch the James Web Space Telescope early.
I thought about the current rovers, but I think they are a bit large to be successful!
Mac users.. (Score:3, Funny)
Mac users everywhere take this as "Oooohh.. there's a Mac inside of those things!" "There are Macs on Mars!" Bleah.
And before you mod me down, realize that I'm an unrepentant Mac user and an Apple Authorized Service Tech.
Monopoly issues? (Score:4, Funny)
Processor is *not* a PowerPC (Score:5, Informative)
No, they're not.
The processors in MER are RAD6000 [baesystems.com]'s, which are radiation-hardened versions of the RS/6000, the predecessor to the PowerPC (see this [baesystems.com] for details). The RAD6000's younger brother, the RAD750, is indeed a rad-hardened PowerPC.
As an aside, there is a big difference between a radiation-shielded processor and a radiation-hardened processor. Shielding implies just sticking some kind of rad-absorbent material between the processor and the environment. A rad-hardened processor is actually manufactured in a different way - different gate layout, different design rules, often different materials (Silicon-on-Insulator is popular). These things are done to minimize or prevent the effects of single-event upsets (when a bit is flipped by high-energy particles) and single-event latchups (which basically turn a couple of gates into a glorified short-to-ground). The materials changes may also improve the overall total dose tolerance of the processor. The work required for redesign is one of the reasons that space-qualified rad-hard processors lag the commercial market. The NASA Office of Logic Design [klabs.org] has some good papers on space processors available online if you're interested in learning more.
Re:Processor is *not* a PowerPC (Score:5, Interesting)
Ask Slashdot? (Score:4, Funny)
"If they ask, we come," Blackman said, echoing the enthusiasm of BAE officials. "I think when something like this happens, the whole community responds to it."
Maybe they should put up an image of the data, the diagrams of the systemboard, and descriptions of the symptoms, and submit it to Ask Slashdot. I'm sure there are a couple of VxWorks experts here who'd love to take a crack at this :)
Had to remove some parts... (Score:3, Funny)
Thanks.
Sincerely, Marvin & Family
No, not PowerPC... (Score:3, Informative)
RAD6000 [baesystems.com]
FPGA's (Score:4, Interesting)
Xilinx radiation-tolerant Virtex(TM) FPGAs are being used in the "main brain" of the rover vehicle, controlling the motors for the wheels, steering, arms, cameras and various instrumentation, enabling the vehicle to travel about the planet.
They also controlled the Pyrotechnical stuff during landing.[Disclaimer: I work for this great company.]
RAD6K (Score:5, Informative)
Re:RAD6K (Score:5, Informative)
Or you fly it as a non mission critical experimental payload which is what we did back in the day I worked on avionics. You fly it as an experimental package so it gets the stress but if it breaks either its not in the mission critical loop, or if it is in the loop you can switch back to proven hardware. I kind of assumed this would be a standard part of qualifying electronics for space flight as well though its obviously a lot more expensive. Its not feasible to test it on Mars due the expense but you could test it in geostationary orbit where it will get lots of radiation and temperature extremes, as well as launch vibration and G's.
make a firewall? (Score:3, Funny)
.
"I wonder if I could make a nice firewall with one of these for my home network..."
1. build computer
2. drop off on mars on way to grocery
3. laugh in face of hacking
4. reconsider when computer mistaken by rover for new type of rock, drilled for mineral content
Mission control software is in Java (Score:3, Interesting)
As noted in a previous slashdot posting [slashdot.org], the software in the control room was written in Java.
A ZDNet [com.com] article says Java made communicating between multiple software pieces very flexible and James Gosling, inventor of Java, spent considerable time helping develop the system. Sun [sun.com] also describes how the same application was used for the Pathfinder mission back in 1997.
The CPU is POWER, not PPC (Score:4, Informative)
The RSC design played a key role in bringing Apple and Motorola together with IBM to create the PowerPC line of CPUs. The 601 was the first PPC and was basically a redesign of RSC. It supported both POWER and PPC architectures, although there were deviances from PPC since the architecture was actually being defined at the time we were working on the chip.
The RAD6000 version of the design happened because IBM wanted to pursue some government contracts, so had the RSC specially qualified. Another group then took the design and performed the radiation hardening.
After Pathfinder we had some cool IBM/Mars posters hanging around the building, but oddly enough they vanished very quickly...
Wrong!!! (Score:2)
Re: (Score:2)
Re:But The Question is.. (Score:2)
Re:But The Question is.. (Score:3, Informative)
it is not uncommon on so hard real time system to disable processor cache, it makes the processor slower, but the response time to an interrupt is easier to calculate. In RTOS interrupt latency must be PROVEN not to be longer than a constraint.
Re:But The Question is.. (Score:3, Insightful)
Re:But The Question is.. (Score:3, Insightful)
Re:hummm (Score:2)
Nah...that would be silly.
Re:hummm (Score:2)
Re:I hope the flash memory was not commodity hardw (Score:5, Interesting)
And the flash memory has probably not failed. It seems to have been a software problem, not hardware.
Rootbear
space shuttle uses 1969-vintage ibm 360 computers (Score:5, Interesting)
Re:space shuttle uses 1969-vintage ibm 360 compute (Score:3, Informative)
I'm not sure which computers this article is referring to, but it may have been an earlier revision of the computer system in the B-52.
The only other computer system I can think of in the B-52s was the ECM system, and since i
Re:20 MHz is not enough (Score:3, Interesting)
Re:20 MHz is not enough (Score:3, Funny)
Post a link to your website and then we'll reconsider your statement
Re:NASA is run by idiots... (Score:4, Informative)
A) They don't need to play Doom 3 up there. 20MHz is sufficient for almost anything you would want to do on Mars. Why send up more than you need to?
B) Your computer runs far hotter and consumes far more power than the Mars rovers do. Power is at a premium when you're millions of miles away from the nearest electrical outlet.
C) The rovers are radiation-hardened. Your system is not. Your computer would last about twelve minutes in space before it locked up. A big part of radiation hardening is using larger (and therefore slower) transistors.
D) It takes years to certify a particular piece of equipment as spaceworthy. NASA isn't going to just pop in the latest and greatest Athlon and assume it will work "because the last one did". That means that anything flying into space is automatically going to be at least a few years behind the curve.