What's Inside the Mars Rovers 458
Captain Zion writes "Space.com has a story about the hardware and software of Mars Rovers Spirit and Opportunity. Basically, they're radiation-shielded, 20MHz PowerPC machines wirh 128Mb RAM and 256Mb of flash memory, running VxWorks. I wonder if I could make a nice firewall with one of these for my home network..."
Re:What's the bus speed on that thing? (Score:5, Informative)
A processor of any speed doesn't need RAM of any size.
The application you want to run needs both processing power and memory. How much of each? Depends on the application.
Comment removed (Score:4, Informative)
Units, units, units!!! (Score:3, Informative)
Mb = Megabits
MB = Megabytes.
The article writes out megabytes, so MB should be used, not Mb!
Re:Radiation Shielding (Score:5, Informative)
Re:Radiation Shielding (Score:2, Informative)
They have extra shielding on the outside, and the electronics on the inside are designed to disipate sudden charges created by radiation hits.
Flying VxWorks to Mars (Score:5, Informative)
Flying VxWorks to Mars [colorado.edu]
Re:Radiation Shielding (Score:3, Informative)
Re:Ouch (Score:4, Informative)
But try that on an OS for a desktop system, and your email program just may blow up your paint program... (remember Windows 3.1's stability? Make that 10x worse)... You can't use VxWorks for the desktop as Windows is used today... it needs a lot of protection... The ease of upgrading is due to the lack of protection...
memory overflow caused Spirit shutdown? (Score:3, Informative)
The lockup happend just as they were going to drill into the rock they've been sitting in front of for nine days. Perhaps there was drill issue too. When the rover memory crashed, it tried to reboot its computer at least a hundred times.
Not 20Mhz (Score:2, Informative)
Re:But The Question is.. (Score:3, Informative)
it is not uncommon on so hard real time system to disable processor cache, it makes the processor slower, but the response time to an interrupt is easier to calculate. In RTOS interrupt latency must be PROVEN not to be longer than a constraint.
Re:Self-warming (Score:5, Informative)
Radioactivity is NOT radioactivity when you are considering things like this. Saying the people who don't want nuclear powered rockets should hate this as well or else they are hypocrites is tantamount to saying that the people who don't like oil spills should bitch about how some motor oil ALWAYS stays in the plastic container it is shipped in. Not quite the same problem. Afterall, these things aren't much more radioactive than a Coleman lantern wick or a smoke detector element...
Re:Redundency Check? (Score:5, Informative)
The online GPC's each carry out the same set of calculations (potentially each uses code designed to do the same thing, but written by different programmers), and they compare each others results. If any single GPC is considered to be too far wrong, the offline GPC's submit their answers. The three GPC's that are in closest agreement then become the new online GPC's, and the remaining two go offline. The GPC's can reboot themselves if they are too far out of whack, if they fail in one of the "results elections", and of course when they are told to do so by the crew.
Also, whenever a GPC is sent offline by one of the others, a specific caution indicator (and potentially the master caution indicator and klaxon) is activated, and the relevant error codes are shown on one of the forward CRT's. The error codes, along with other information such as the currently running program and the current mission phase, determine the crew's actions. Actions can be as simple as disabling the master caution klaxon for the current alert, all the way to hand-checking certain results and manual GPC restarts.
This is all from memory (from about 5 years back), so some of this may have changed recently, particularly on Atlantis with the "glass cockpit" upgrade that happened 18 months or so ago, but the general gist should be about right (and I'm sure I'll soon know if it isn't!!)
Re:Wait a second... (Score:5, Informative)
Go to
http://www.iews.na.baesystems.com/space/rad60
and click on the rover picture to get a PDF brochure, which gives the 33MHz/35MIPS figure.
Rootbear
Re:memory overflow caused Spirit shutdown? (Score:2, Informative)
The first intergalactice filesystem bug
Processor is *not* a PowerPC (Score:5, Informative)
No, they're not.
The processors in MER are RAD6000 [baesystems.com]'s, which are radiation-hardened versions of the RS/6000, the predecessor to the PowerPC (see this [baesystems.com] for details). The RAD6000's younger brother, the RAD750, is indeed a rad-hardened PowerPC.
As an aside, there is a big difference between a radiation-shielded processor and a radiation-hardened processor. Shielding implies just sticking some kind of rad-absorbent material between the processor and the environment. A rad-hardened processor is actually manufactured in a different way - different gate layout, different design rules, often different materials (Silicon-on-Insulator is popular). These things are done to minimize or prevent the effects of single-event upsets (when a bit is flipped by high-energy particles) and single-event latchups (which basically turn a couple of gates into a glorified short-to-ground). The materials changes may also improve the overall total dose tolerance of the processor. The work required for redesign is one of the reasons that space-qualified rad-hard processors lag the commercial market. The NASA Office of Logic Design [klabs.org] has some good papers on space processors available online if you're interested in learning more.
Re:What's the bus speed on that thing? (Score:3, Informative)
Disclaimer: Even faster flash devices may exist, but I don't know about them.
Re:Self-warming (Score:5, Informative)
Radioisotope heaters use much less material, as they only need enough heat to keep the warm electronics box above -40F or so. From the Environmental Impact Statement in the Federal Register ([wais.access.gpo.gov][DOCID:fr10de02-54]):
"Each rover would employ two [calibration] instruments that use small quantities of cobalt-57 (not exceeding 350 millicuries) and curium-244 (not exceeding 50 millicuries) as instrument sources. Each rover would have up to 11 RHUs that use plutonium dioxide to provide heat to the electronics and batteries on board the rover. The radioisotope inventory of 11 RHUs would total approximately 365 curies of plutonium."
Nothing you'd like to swallow, but still, much smaller than a radioisotope power unit.
No, not PowerPC... (Score:3, Informative)
RAD6000 [baesystems.com]
They used to use sapphire... (Score:5, Informative)
...as the substrate of the chip, rather than a silicon wafer, so the chip was a "sapphire" chip rather than a silicon chip (although doped silicon could then be used to form transistors, as could Gallium Arsenide or Germanium, through the regular lithographic process).
This is the classic "Silicon On Insulator." IBM has a process of embedding a layer of glass beneath the surface of a standard silicon wafer, allowing SOI using silicon substrates. This and their work with copper set them apart from the other large silicon transisitor foundries (TSMC, Intel, etc.).
The processors on the rovers are probably SOI, but I don't know which process is used.
Re:What's the bus speed on that thing? (Score:3, Informative)
RAD6000s seem closest to PPC601s (Score:4, Informative)
By all other standards however, they seem to be closely related in time and technology to the 601, which powered Powermac 6100, 7100, 8100, 9150, 7200, 8200 and 7500 PPCs, as well as I think, one of IBM's thinkpads.
Those 601s are a very nice chip, and quite underestimated at what they can do at low clock speeds. If the RAD6000 is anywhere similar, I can understand why it was picked.
RAD6K (Score:5, Informative)
Re:Wait a second... (Score:3, Informative)
Re:memory overflow caused Spirit shutdown? (Score:2, Informative)
Re:RAD6K (Score:5, Informative)
Or you fly it as a non mission critical experimental payload which is what we did back in the day I worked on avionics. You fly it as an experimental package so it gets the stress but if it breaks either its not in the mission critical loop, or if it is in the loop you can switch back to proven hardware. I kind of assumed this would be a standard part of qualifying electronics for space flight as well though its obviously a lot more expensive. Its not feasible to test it on Mars due the expense but you could test it in geostationary orbit where it will get lots of radiation and temperature extremes, as well as launch vibration and G's.
Re:Radiation hardness (Score:4, Informative)
Swapped out? To where? You're being a bit recursive here. The operation failing is the initialization and reading of the flash. You can't swap to flash while you're initializing it.
However, I doubt they implemented virtual memory on the thing. It's far too much overhead in the OS to actually implement paging, etc. They were probably just very, very careful about the amount of memory in use at any one time, as with most embedded systems. Someone, however, didn't check that in a massively fragmented flash filesystem, the directory read wouldn't take up the entire RAM. Oops.
Re:Wait a second... (Score:3, Informative)
danheskett: That's an excellent point. A lot of people are thinking instruction = 1 cycle. The real world is that it's not unusual for an instruction to take 2, 4, 10, or even 100 cycles. The reality of the matter is that instructions can be anything from a single two bit sum to a floating point division. I see this mistake a lot...
You both assume that the one who wrote the article didn't make the same mistake in the opposite direction.
In this article about the Stardust probe [spaceflightnow.com], the RAD6000 is said to be "a radiation-hardened version of the PowerPC chip used on some models of Macintosh computers" which "can be switched between clock speeds of 5, 10 or 20 MHz".
The CPU is POWER, not PPC (Score:4, Informative)
The RSC design played a key role in bringing Apple and Motorola together with IBM to create the PowerPC line of CPUs. The 601 was the first PPC and was basically a redesign of RSC. It supported both POWER and PPC architectures, although there were deviances from PPC since the architecture was actually being defined at the time we were working on the chip.
The RAD6000 version of the design happened because IBM wanted to pursue some government contracts, so had the RSC specially qualified. Another group then took the design and performed the radiation hardening.
After Pathfinder we had some cool IBM/Mars posters hanging around the building, but oddly enough they vanished very quickly...
Re:NASA is run by idiots... (Score:4, Informative)
A) They don't need to play Doom 3 up there. 20MHz is sufficient for almost anything you would want to do on Mars. Why send up more than you need to?
B) Your computer runs far hotter and consumes far more power than the Mars rovers do. Power is at a premium when you're millions of miles away from the nearest electrical outlet.
C) The rovers are radiation-hardened. Your system is not. Your computer would last about twelve minutes in space before it locked up. A big part of radiation hardening is using larger (and therefore slower) transistors.
D) It takes years to certify a particular piece of equipment as spaceworthy. NASA isn't going to just pop in the latest and greatest Athlon and assume it will work "because the last one did". That means that anything flying into space is automatically going to be at least a few years behind the curve.
Re:space shuttle uses 1969-vintage ibm 360 compute (Score:3, Informative)
I'm not sure which computers this article is referring to, but it may have been an earlier revision of the computer system in the B-52.
The only other computer system I can think of in the B-52s was the ECM system, and since it was highly classified, and I did not work on that system, I'm not overly sure.
For a little extra karma whoring, i'll relate a funny story. Our base, which shall remain nameless, would fly early morning ECM sorties while people were driving into work.
Their target signals for jamming? The radar guns the cops used in their speed traps to try and catch our folks driving into work. Apparently, the cops finally caught on and complained at the high number of equipment failures with their radar gun equipment.
Re:Radiation hardness (Score:2, Informative)
BTW: VxWorks 5.5 does support FAT32.
--Robert
Re:Radiation hardness (Score:1, Informative)
If you are allowed to pour you hart out about how vxworks sucks for your router/broadband project then can I offer some clues and guesses as to why NASA might have chosen it?
The article stated that NASA needed to be able to patch code while it was running, a very reasonable demand as one cant just get the rocket back to earth and start the whole thing over.
Qoute: "In addition to VxWorks' reliability, the system allows users to add software patches -- such as a glitch fix or upgrade -- without interruption while a mission is in flight. "We've always had that [feature] so you don't have to shut down, reload and restart after every patch,""
Iirc from the nat. geographic documentaries the spirit code was still being developed while the rover was on its way to the planet. (How is that for missing a deadline, they must have learned this one from the game industry ;-) ) I may be mixing things up with the orbiter though. This would be a perfect explanation as to why there apears to be a bug/situation which would have been found with running long tests collecting lots of "science data" on earth.
Now you go on about the vxWorks malloc implementation (dynamic memory managment features). NASA may actually have little need for having the OS manage the memory. It apears most of the colected data is stored in a filesystem on flash ready for pickup by a comunication system sending it back to earth (By any available orbiter). Now the problem with fragmentation is that it is likely to get worse as soon as it starts, but with 128 MB of ram and *very* few procecsses I dont see how you could get a problematic or fatal amount of fragmentation unless there are multiple procecses collecting a lot of data at the same time in very high resolution mallocinging tiny amounts (less then a page) at a time with little memory left to spare. Neither of these conditions seem to be likely to be met in any big way, but I havent seen this codee ofcourse.
Now you mention tcp/ip is bad, and I fully believe you. (eventhough my motorola cable modem claims to be running vxworks, consumer broadband must be at least possible with this rtos ;-) ). But, obiously, NASA is not gona use tcp/ip as a protcol and the drivers for the radio hardware are not gona be a standard part of vxworks. I think these pieces of code for sending stuff over high frequency duplex radio data links have been laying around at nasa (or defence contractors) tested and proven on many applications just like many other components of this machines software. This bring me to my other guess as to the choicec of vxworks, the rover has to pretty much plot its own course over the planet. It seam possible that the rover now on mars uses the excact code for that that early prototypes used while spending months in american deserts. VxWorks might have been the os of choice by the prototye engineers, but porting this code to another OS might introduce bugs which goes for all the code nasa already had (tested, and perhaps even proven in space)
Now unlike VxWorks I have run QNX, (as has everyone who has had the time to go over here [qnx.com], have a look If only to brag about using another os on your desktop). While I agree that it is a very very nice os, I could very well see it not having any features over vxworks that would make it worth it for nasa to port all their current vxworks stuff over only to have to test it again while still not knowing for sure if it will run *excactly* like on vxworks. And ofcourse any features vxworks did lack (or rather did have while not being supposed to ;-) ) could have been fixed by nasa coders in no time just like you did only with the help of the source.
Re:Radiation hardness (Score:3, Informative)
consider the use of malloc()/free(). In an RTOS it does not much sense to dynamically allocate and free memory at run time. Same thing applied to other sytem resources, like tasks/processes. This takes quite a bit of time from a Real Time point of view to create new tasks, semaphores, objects, memory allocation, etc.. Most RTOS engineers, who know what they are doing, pre-allocate all the memory they need at boot time, start all the processes they will need, etc. In order words you allocate all your system resources at boot time so you keep your real time characteristics. An RTOS system is static. no dynamically process allocation, memory allocation (i.e. unexpectantly running out of memory), etc. You start everything up and account for your needs. A very different model from probably what you are used to in a Desktop/Server OS. Think about it as static appliance. Give me one good reason why you can't preallocate memory and all the tasks you will need?
Yes memory protection has its benefits. Speed is not one of them. Remember this is an RTOS. How many RTOS do you know of that support memory protection? hmm.. 1 I believe. Which is VxWorks AE. Which you say is slow. gee. If you want a low latency fast RTOS, memory protection quickly falls off your "must have" list.
So VxWorks supports FAT16. So that makes it suck?
It also supports FAT32, NFS, a Flash Filesystem, etc. So all those points make it such too. I see your reasoning.
VxWorks, every other true RTOS, does not use the system clock exclusively to multitask a you say. In fact, round robin multitasking which uses the system clock is turned off by default Unlike a server/desktop OS, a context switch is not generated strictly via an interrupt (system clock, device interrupts). context switches can occur at any time as your code generates events. For example, giving a semaphore to unblock a test, sending message to a message queue which unblocks a waiting receiver, trying to get hold of an object which is in use (this would block you). This does not require a high latency and it almost immediate. This is because all code runs in processor supervisor mode. i.e. no traps to communicate from one task to another. This goes along with the no memory protections reasoning. It has benefits. Let me guess you didn't read the manual or take a vxWorks 101 class, right?
Nasa has been using vxWorks primarly on most of their projects since 1980! I think that says alot by itself. Thank you for your "expert" opinion in helping people judge the quality of an RTOS.
Re:Radiation hardness (Score:5, Informative)
I know we're not the only ones to have been burned by Wind River's malloc. I know several major companies that also had to replace Wind River's code.
As far as being able to dynamically replace code, VxWorks isn't alone in that. Numerous other RTOSes out there can do the same thing, including QNX. QNX even supports the concept of a hot standby process to take over if the main process dies.
To give you an idea about how Wind River's malloc works, they keep a sorted linked list of fragments from the smallest to the largest. When you try and allocate a block, it walks the linked list until it finds a block large enough. Likewise, when you free a block it checks if it can coalesc the block with a neighboring block. It then goes through the linked list looking for a slot to insert the free block.
Yes, VxWorks may have been around since the 80's, but that's part of the problem too and it is showing its age. In the 80s embedded processors typically did not have MMUs. Now MMUs are quite common in the more powerful embedded processors.
You say you can't have low latency and memory protection? QNX proves that you can. It is low latency and *very* robust. If your driver dies, no problem, restart it. Timesys Linux also has a very low latency, although not as low as QNX. Timesys also has an interesting feature where you can guarantee CPU and networking resources. I can schedule a task to be guaranteed 5.8ms of execution every 8.3ms and it will guarantee that that task will get the CPU time allotted to it with the desired resolution. This is without increasing the system tick rate (usually 10ms). Timesys can also schedule a task to be higher priority than an interrupt. I'm not as familiar with QNXs scheduler, but it's also quite flexible from what I've heard.
As far as FAT, it is not a robust filesystem. It never has been. If the FAT gets corrupted or a directory entry gets corrupted it's difficult to recover. Other than possibly having 2 copies of the FAT cluster table, any corruption can be difficult to repair. If the FAT table gets corrupted, which table is corrupt and which is not? If a directory entry gets corrupted, it can be impossible to fix. For flash memory, unless you are using a device with special wear-leveling, FAT is about the worst choice since any file write that changes the size of a file requires a write to the directory entry and possibly the FAT table. If the table gets corrupted and you don't run a repair operation (which often ends up leaving orphaned files as lost clusters), the file system can happily corrupt itself to death. Why do you think every time DOS/Windows9x/ME crashed it had to repair the disk with scandisk? FAT is a poorly designed file system that was originally designed for 160K floppies and scales poorly. FAT32 is an improvement, but it's still not very robust. For flash, something like Linux's journalling flash file system 2 (JFFS2) [redhat.com]. More information on VxWorks file system support can be found here [xs4all.nl].
Basic VxWorks information can be found http://www.slac.stanford.edu/exp/glast/flight/docs /VxWorks_2.2/vxworks/guide/ [stanford.edu].
Re:Radiation hardness (Score:3, Informative)
QNX has a lot of reliability features built in that are not present in VxWorks, Linux, or RTEMS. It can take periodic snapshots of a task and recover if a task dies.
Also, field debugging of VxWorks can be a real pain. It does not generate a core file you can debug later.
Timesys has very good real time support, but the context switching time is on the order of 10us depending on the processor, QNX is better.
QNX is more more like a unix environment with real threads and processes. QNX relies mostly on effecient message passing between tasks. QNX also has much better TCP/IP support than VxWorks, including support for IPv6. From what I can tell, development and debugging in QNX is much easier than VxWorks. QNX does not have as many priority levels as VxWorks. QNX Neutrino supports 64 priorities. VxWorks supports 256. Timesys Linux can support thousands of priorities.
QNX also has built-in support for distributed processing as long as there is an interconnect. It also supports SMP (VxWorks does not).
VxWorks, on the other hand, is like a single task with many threads. A "task" in VxWorks is more like a thread in that all memory is globally accessable. Any function can call any other function.
I havn't worked with QNX or Timesys personally. I've spent the last 4 years developing for VxWorks Tornado II. The group where I work dealing with OS related issues has more or less decided to use either QNX or Timesys for our next generation software. Timesys looks good from the tools perspective and the wide Linux experience out there, plus the fact that currently our product runs on a mixture of Unix and VxWorks, plus many of the drivers we need are available for Linux and would need to be ported to QNX. Some of the key software we are using has been ported to QNX, though, and the reliability of QNX is also of interest. QNX has also won some nice design wins like Cisco. Since we make routing equipment we tend to pay attention to things like that.
Also, beware that Wind River's support isn't very good from our experience. It basically boils down to "what support?" even with a support contract. Wind River has layed off a lot of people lately and I'm sure that hasn't helped the company. They've been bleeding money like crazy and are hurting pretty bad. They've been losing market share to other embedded RTOSes and Linux. Hell, Wind River has ported their development tools to support embedded Linux (I wonder why?).
As far as real-time Linux is concerned, I looked at several of them. RT-Linux requires that RT tasks talk to the RT kernel and use a separate API and drivers than the Linux tasks. It's basically Linux running on top of a RT kernel. Monta Vista does not provide hard real-time, and when I last spoke with them they had no solution for priority inversion. Timesys modified the stock Linux kernel so RT tasks don't need to use a separate set of APIs from non-RT tasks and Timesys can guarantee timing, both for the CPU and networking. For soft real-time, Monta Vista seems to be pretty popular. The people I spoke with who used it said it was quite reliable but couldn't comment on support since they never needed any.
-Aaron