Houston, We Have a Software Problem

An anonymous reader writes "The computer system that launches the Space Shuttle is an old, but important, computer system. It is built from mid 70's technology and features SSI chips like 7400's...which are getting hard to find. It has 64k of memory and no room to repair any software bugs. NASA started the CLCS project in 1996 which uses state of the art computer languages, OO methodologies, and hardware. Everything that you could actually hire people off the street for. However, NASA is in a budget crunch with the Space Station cost overruns. It is looking to trim costs to keep the Space Station going. There are stories about CLCS getting cancelled here and these guys say its already cancelled."

7400s hard to find?

[Istealmymusic]

I don't know about everyone else, but when I was a kid I got a Radio Shack 300-in-1 electronic project kit for my birthday which came with a dozen or so 7400 chips. When I plugged one in backwards I just went down to my local Radio Shack [radioshack.com] and picked up a new 74LS00, which they had plenty of in stock all the time.

Certainly the 7400 series as a whole is still widespread and used in hobbyists kits, I'm not that old. Maybe the original 7400 is becoming obsolete, being replaced with the 74LS (low-power Schottkey) or CMOS chips? If then it shouldn't be too difficult to replace the TTL logic with CMOS logic, given a few adjustment levels in voltage, or they could use the TTL-logic and CMOS-logic in one compatible chips [cjb.net].

Of course, the 5400 series SSIs (small-scale integrated circuits) are preferred over the 7400s for industrial purposes, and as a plus they are completely backwards compatible. Why isn't NASA using those?

Re:Why not simulate it?

[rodgerd]

Given LISP and (IIRC) Smalltalk both existed in the 70s, the world wasn't as primitive as you make out.

Besides, the use of modern programming buzzwords implemented by college kids sounds like the principal problem with this project...

Re:Why not simulate it?

[perfects]

Given todays hardware, why you can't just simulate the old system

You can't just buy a system from Dell and put it into the Space Shuttle. You can't use a Pentium, a modern hard drive, Linux, Windows, or Open Source anything.

As far as the hardware goes, everything mission-critical that goes aboard the Shuttle has to be ruggedize against incredible vibration, tested a thousand different ways to make sure that it can't be affected by exposure to vacuum/heat/cold/radiation/cosmic rays/etc., tested another thousand ways to make sure it doesn't interfere with other critical Shuttle systems... and on and on.

And a bug in the newly written software could cause not only the death of several astronauts, but potentially the loss of a Shuttle, a launch facility, and the ISS. Would you, under any circumstances, put your life, five other lives, and billions of dollars in the hands of software that you found in an Open Source project?

On your desk a "Fatal Error" isn't, really. But 60 miles up?

Space Computing: Some Numbers

[aebrain]

From an article in the Sydney Morning Herald [smh.com.au].

Only 58 centimetres square and weighing 50 kilograms, the tiny FedSat satellite is packed with five scientific experiments and all of the instruments required to communicate with Earth during its anticipated three-year life. At the heart of the satellite is a 10MHz ERC-32 processor - a SPARC-based 32-bit RISC processor developed for high-reliability space applications.

The ERC-32 sacrifices processing power for durability and reliability. It uses three chips to process a modest 10 million instructions per second and two million floating-point operations per second - less than 1 per cent of a Pentium 4's capabilities.

The pay-off is reliability: the ERC-32 uses concurrent error-detection to correct more than 95 per cent of errors.

Power-hungry microprocessors such as the Pentium 4, which runs a standard office PC bought off the shelf today, would be an intolerable burden on the solar-powered satellite. The ERC-32 consumes less than 2.25 watts at 5.5 volts.

Designed to survive extreme radiation bursts from solar flares, the ERC-32 can tolerate radiation doses up to 50,000 rad. This is 100 times the lethal dose for humans.

...A team of Australian programmers developed FedSat's onboard software, building on work done in Britain. It is written in Ada-95, a programming language designed for embedded systems and safety-critical software. All it has to work with is 16MB of RAM, 2MB of flash memory for storing the program, a 128K boot prompt and 320MB of DRAM in place of a hard disk that would never survive the launch process. All essential data is stored in three physically different locations.

The software is built in a similar way - lots of internal checks, tell-me-thrice memory, soft-failure-bit-flip-correcting daemons etc. In this case, lives aren't at stake, but the people doing the programming are used to situations where they are.

Re:They should make it open source

[Anonymous Coward]

They could always look here

http://flightlinux.gsfc.nasa.gov/ [nasa.gov]

maybe a case of the left hand not knowing what the right hand is doing

The only reliable piece of the Shuttle

[HuguesT]

At the time of the Challenger inquiry, the late physicist Richard Feynman was part of the investigation committee. He found that most of NASA at the time was in full delusional mode about how reliable the Shuttle really was.

The only exception was the computer systems group, in particular the software side. They had metrics, procedures and rigour.At the time of the enquiry the hardware was already old.

It's the attitude that counts, not the hardware, not the methodology of the month. OO is not going to solve NASA's problem, it's going to be difficult. Myself I'd just make sure that the hardware would always be available, and not change a thing.

Re:7400s hard to find?

[rant-mode-on]

• When I plugged one in backwards I just went down to my local Radio Shack and picked up a new 74LS00

Dunno about the Shuttle, but I assume my experience applies. I used to write autopilot & autostabilser software for helicopters. They used 80286 & 68000 CPUs, which have started to become more difficult to find. Not because there are no 286's or 68K's out there, but because there aren't so many 286's and 68K's available that are certified for flight.

Re:Correction...

[blincoln]

The Russians *have* shuttles, but they never used them beyond tests.

Buran -- Russian for snowstorm -- is the name of the Russian shuttle that made one unmanned spaceflight in November 1988.

It circled Earth twice, landed automatically and since then has sat in storage at the Baikonur Cosmodrome in Kazakhstan. On May 12 the vehicle was damaged by falling debris when portions of the roof of the building the Buran was in collapsed.

Several other copies of the Russian shuttle were built as part of a test program and through the years have all become known by the name Buran.

Of those, one Buran was turned into a space-themed restaurant at Gorky Park in Moscow and another was given a fresh coat of paint before going on display for more than a year in Sydney, Australia during the time of the 2000 Summer Olympic Games.

(from a Space.com article [space.com])

IIRC, they swiped the design from the US shuttle, and made some... odd modifications like leaving out the main engines.

80's technology was the best

[XNormal]

I used to work with military electronics and found that the best gear was always from the 80s. The stuff from the 60s and 70s (yes, some of that is still in service) was too primitive. The 90s hardware was too complicated and suffered from unreliable software.

In the 80s the microcontroller technology was just good enough to embed a processor with 64k of ROM full of finely crafted code written by a single programmer and it always just worked, perfectly, every time.

Re:Space Computing: Some Numbers

[aebrain]

The context was that of software for an unmanned microsatellite, not the shuttle.

Crewed spacecraft have an even more strict set of rules attached to the software development process. Have a look at some of the articles [af.mil] on DO-178B [lynuxworks.com], the software development standard for avionics. Similar issues apply, but even more so.

Look, people - not Geniuses - just normal, everyday programmers - have been making software you can bet your life on [adaic.com] for a long time now. We know how to do it even more cheaply [af.mil] than the normal buggy commercial work (though testing is radically expensive and blows out the total cost). There's no need, and no excuse, for BSDs and security problems. None. You just have to have the right tools, the right training, and the right attitude. If you like, the Right Stuff [fastcompany.com]. Here's a quote from that article:

It's strictly an 8-to-5 kind of place -- there are late nights, but they're the exception. The programmers are intense, but low-key. Many of them have put in years of work either for IBM ( which owned the shuttle group until 1994 ), or directly on the shuttle software. They're adults, with spouses and kids and lives beyond their remarkable software program.

That's the culture: the on-board shuttle group produces grown-up software, and the way they do it is by being grown-ups. It may not be sexy, it may not be a coding ego-trip -- but it is the future of software. When you're ready to take the next step -- when you have to write perfect software instead of software that's just good enough -- then it's time to grow up.

People like myself look upon any work over about 7 hours a day more than twice a month as signs that "I personally screwed up", because I'm the guy who sets the schedule, not some PHB. We have lives. We have kids. We have hobbies. And the stuff we do is hard, the systems do a lot more than most commercial apps, and with far fewer memory and CPU resources. It's both incredible fun "boldly going.." and all that, but also a crushing responsibility when we do safety-critical work. People's lives depend on us doing the best possible job we can.

One area I disagree with in the "Right Stuff" article is that the work doesn't involve creativity. This is balderdash - we're doing stuff no-one has ever done before under really tight resource constraints. To get a reliable architecture often requires significant smarts, lateral thinking. Anyone can make a complex solution to a complex problem, the really good guys and gals make solutions so drop-dead simple, obviously-correct and efficient that it's miraculous how much such simple, obvious and readable code actually accomplishes.

Looking at the general world of InfoTech, we see that most programmers out there would rather write the winning entry for the "Obfuscated C" contest than make some software that gets us around the solar system. And that people who make reliable software hit the unemployment queue on project completion, while those making buggy stuff have jobs-for-life in maintenance. Of course, they often have 80-hour weeks too, and are driven by PHBs who know b* all, and can't even take pride in the product, so there is some justice.

Re:using GNU software, too

[aebrain]

You're correct, GNAT 3.13p. Anyone with mod points, please give this guy one for "Good Deduction"