Why ISS Computers Failed

Geoffrey.landis writes "It was only a small news item four months ago: all three of the Russian computers that control the International Space Station failed shortly after the Space Shuttle brought up a new solar array. But why did they fail? James Oberg, writing in IEEE Spectrum, details the detective work that led to a diagnosis." The article has good insights into the role the ISS plays as a laboratory for US-Russian technology cooperation — something that is likely to be crucial in any manned Mars mission.

Re:Interesting hardware problem

[WindBourne]

Russia has shown that they do not consider humidity to be an issue. In particular, the MIR was all but finished because it had mold everywhere.

Russia taught us a lot about space construction and staying alive in a space station. But likewise, we have also done the same. But it is obvious that there is room for more growth.

Power off command

[jsse]

Also, in a shocking design flaw, there was a "power off" command leading to all three of the supposedly redundant processing units.

That reminds me many years ago, when my friend worked as a programmer in a major bank writing small programs for an online international financial system. He issued an 'shutdown' command through JCL(Job Control Language) and that really shutdown the entire system. He didn't realize he had the privilege to issue administration commands. Instead of reporting the crisis to his manager, he hide away until someone figured out what's going on. Needless to say, my friend was fired.

Years later I met his manager, he told me that my friend could have been promoted for discovering one of the biggest loophole ever in the bank's history, if he had reported the problem immediately. Though the unexpected shutdown caused considerable damage, it could have saved billions from real break-in with this loophole.

That's a lesson that every engineer should have been learned. :)

Cascading failures

[j-stroy]

True, as a starting point.. Tho, failures tend to be things that snowball. Its sort of an anthropic principle [wikipedia.org] of failures. ie Bad things happened because failures were happening.
I have always tried to learn from air crash investigations and so on how failure modes develop. In problem solving mode, it seems one should assume the distinct possibility of multiple problems all at once.
In this case, multiple failure paths existed, tho it took a power spike to set it off as I interpretted it. Even without corrosion, it seems the system would have failed, though not irrecoverably.
I repeatedly ask the question "Is that everything? Is there anything else that could come from that?" It seems the engineers didn't perform enough diligence on the trickle down effects.

Re:Urgh.

[Jugalator]

I agree... That's what first came to mind after having watched this incident unfold live. What he fails to mention is that the Russian engineers were always open to suggestions and they cooperated pretty well when they needed to discuss the problems. The Russians were also working nearly 24/7 on trying to find and resolve the problems and come up with theories before they were running out of time. The article makes it sound like they early on got locked into "blaming the Americans" or something. It was merely one theory that was tossed around and discussed, and diagnosed early on. If there seem to be a power failure (which it ended up being all about), surely one logically suspected culprit could be a power feed problem?

Re:Urgh.

[jamstar7]

You're thinking Levi Strauss. Leo Strauss was the inspiration for the NeoCon movement.

Re:A bit harsh on the Russians.

[jamstar7]

I'm thinking it's relatively close to even. We lost 3 on the pad (early Apollo, where we learned that a full oxygen mix in a capsual with burnable stuff in it is Almost A Good Idea), & a pair of crewed space shuttles. Officially, the Russians haven't lost anybody but rumor around the water cooler is, they lost a couple when they couldn't deorbit a capsual in time and the cosmonauts ran out of oxygen, couple died on the pad in explosions, and a couple parachute failures pancaked a couple Vostoks into the Siberian tundra.

Re:Hmmmm.

[WindBourne]

Problem with doing the small lift, is that the ISS would have been a fraction of the size that it is. Until they developed transhab, each module would have to be rinky dink.

Personally, I would argue that not moving forward on new lifters was THE real mistake. In particular, during reagans time was when the Challenger happened. reagan should have started the development on a new lifter then. Clinton did start one (X-33), but it was killed off with W. Right now, I would have to say that if America can get multiple launchers that can lift 25 metric tones inexpensively AND perhaps 2 launchers that are true Saturn class (the Ares IV|V and the the falcon BFR), then we would be ok for some time, perhaps 2020-2025. What amazes me is that we expected a new class of rocket to last like an airliner. Yet, Rocket Science is in the same place that Airplanes were in the 40's; roughly undergoing all sorts of changes due to loads of new research. Hopefully, we learned from all this.

Re:The REAL reason they failed

[sortius_nod]

Vista doesn't get any breaks for 1 reason: I run Vista and it runs GREAT! BUT... big BUT. I am currently running: Intel Q6600@3Ghz 2x 1GB 800Mhz RAM 2x ATi HD2600 XT 256MB GDDR4 P35 Chipset Motherboard. Runs REAL smooth. I tried to run it on my old AMD 4000+ X2/2GB/ATi x1950. I had to pretty much scale it back to almost look like XP before it was anywhere near usable - and why would I do that, when I have XP. I know this is off topic. To bring it back to topic, nice read of an article, it seems the ISS is prone to the same problems we have down here - interoperability.

Re:The REAL reason they failed

[KDR_11k]

From what I read vacuum tubes are immune to EMP.

Re:The REAL reason they failed

[The Spoonman]

Why? Is defending a MS operating system for honest reasons impossible to believe anymore?

Here's the problem: the vast majority of Slashdotters are either: a) technically incompetent or b) Unix people, which also makes them technically incompetent but also gives them an unjustified superiority complex. After all, their OS of choice has gotten to the point that they have to assemble it themselves and then give it away for free. And despite all of that, people still don't want it. Go fig.

In all seriousness, they just don't get it. It's a shame, and it's just getting worse every day. The industry's filled with old farts who refuse to learn anything new, and young'ins with no aptitude beyond passing a certification test. When I tell them our team of 15 people manage 14,000 Windows desktops and 2000 Windows servers, they tell me it's impossible. But, again, that's 'cause they're boobs. Trust me, just keep pissin' 'em off by showing them up in projects and eventually they start to dwindle away.

Re:The REAL reason they failed

[Rycross]

On the flip-side of the coin, the only problem I've had with Vista is that two of my games didn't work: one due to bad drivers, and another due to their invasive anti-cheat software not playing well with a 64 bit operating system. A lot of coworkers and friends who have tried it have similar experiences: that is, they seem to get on fine with it. When all the experiences I come across are positive, and the only negative experiences related are on blogs or sites with bias, it makes it seem like FUD.

Of course, the reality is that a lot of people have problems, but a lot of people run it just fine. But that second group of people isn't represented well, and if people try to step up and represent them, they're loudly accused of being paid shills. And frankly, I'm willing to go out on a limb and say that the accusers have no intellectual integrity, since they'd rather resort to ad-hominem instead of considering the other person's experience.

Re:The REAL reason they failed

[Blakey Rat]

Not necessarily. But when we read about numerous Vista show-stoppers, it's hard to believe that some people are having a good experience with the OS. We might conclude that these folks don't push the OS very hard, so the flaws are not exposed. Other conclusions are possible -- maybe it really DOES work for some people.

Or maybe the "show-stoppers" you're hearing about are nothing but pure weapons-grade bullshit in the first place.

Underneath it all, many people are waiting for MS to release "a better Unix than Unix". Until they do, people will be quick to side with Linux as the better choice. Apple made the big jump with OSX; time for MS to do the same.

They already have! Windows Vista has a better permissions system than Unix, it's equally stable, it's got a great new CLI environment with Monad (or whatever they're calling it these days), it's capable of running a variety of apps no Unix system ever dreamed of. It works with hardware that no Unix system ever dreamed of, until Windows came along and said "hey, let's do this." (Do you think a purely Unix world would ever have tablet PCs or webcams?)

I think you have the challenge reversed. The challenge is for Linux/Unix developers to create a product that does everything Windows NT-based OSes do, but better. But the actual goal of Linux/Unix seems to be to do the bare minimum to keep up GUI-wise, but only as long as your 1974 CLI scripts still run. All the major open source apps, except Firefox and Apache (and perhaps a few others) are at least 5 years behind the competition, even the competition that's not even 5 years old, like Apple's iWork suite.

Re:Urgh.

[DerekLyons]

I don't know if the Russian Program Managers got all political against us, but the item, written by a retired NASA manager, sure as hell gets political against the Russians.

When you follow the space progam/ISS day in and day out, rather than relying on the all to infrequent Slashdot coverage... you soon see why. Again and again when something goes wrong, the Russians first (publically) announced 'theory' is that the problem is 'the Americans fault'. Only months later, if ever, does the truth come out. There are a couple of failures from the early flights of the current Soyuz version that were publically blamed on the Americans - that the Russians have yet to disclose the real cause of. The Russians have a long habit of being less than candid when it comes to their space program, and NASA has gone right along with them in covering up safety and performance issues with MIR, Soyuz, and the ISS.
 
 

This item is hugely biased. It looks to me like a simple case of corrosion, which could easily have been patched up if it happened on a Mars flight.

Sure, this one failure could have been patched up - but this is only the latest in a long series of failures caused by poor design and manufacture of the Russian segments of the ISS. Failures nowhere matched on the US side. Failures consistently blamed on the US by the Russians. While both NASA and the Russians are publically praising the performance of the Russian hardware.
 
It's not just about the Russians.
 
 

It's just as well he's retired - looks like he's fighting long lost battles against cooperation with the Russians and Europeans.

It may seem that way to somebody unfamiliar with the backstory and history. (I.E. pretty much every Slashdot commentator so far.)
 
[rant]The Slashdot hivemind frustrates the hell out of me when it comes to space issues. Too damm few bother to actually read and keep up with the field, and fewer still know much about the history.[/rant]