ISS Computer Failure 289
A number of readers wrote us with news of the computer problems on the International Space Station. Space.com has one of the better writeups on the failure of Russian computers that control the ISS's attitude and some life-support systems. Two out of six computers in a redundant system cannot be rebooted. The space shuttle Atlantis may have its mission extended until the problem is fixed. A NASA spokesman was optimistic that the problem can be resolved; worst-case scenario would be for the shuttle to evacuate everyone onboard the ISS. Engineers are working on the theory (among others) that the failure may have been triggered by new solar panels installed earlier in Atlantis's mission.
Re:How bad a worst case? (Score:2, Informative)
If they need to evacuate, there are sufficient Soyuz escape modules (tried and tested as the standard re-entry module used by Cosmonauts for the last 40+ years with an almost unchanged design) for all of the current crew capacity on the ISS. Well, I hope so for there sake, or we might have a spaceborn version of what happened to the unfortunate inhabitants of the S.S. Titanic, where passengers vastly outnumbered available spaces on the lifeboats of the supposedly unsinkable ship.
Re:Does the ethnicity matter? (Score:3, Informative)
This [amazon.com] is an interesting read on this subject. The answer to your question is that the fact that the computers are Russian probably does matter.
It's not that the Russian mathematicians aren't excellent, it has more to do with their engineering approach.
That, and of course politics on both sides...
Re:Hey, here is a crazy idea (Score:5, Informative)
The investment in time, money, and energy has already been made. To abandon it now, no matter how dysfunctional it is, would be a bigger waste. If the initiatives to return to the Moon and move on to Mars are going to go forward (and given Congress' past performance in this regard, I highly doubt it), then ISS is a necessary platform to span the gap between the Earth and the Moon. MInd you, when the United States was first thinking of going to the Moon, Werner von Braun put forward the plan to build a space station first, then use it as the assembly point for the journey to the Moon. Then, the platform would already have been established, and the Space Shuttle would have been the next natural extension after the end of Apollo. But the idea was shelved in order to get to the Moon by 1970, and as a result we have the current situation. So, we have done it backwards, but to abandon it now would be truly a giant step in reverse.
definition of attitude (Score:5, Informative)
Maybe the new solar panels are a new input to the attitude program - "I am a new solar panel, I need to be pointed this way so that my 1 axis motor can track the sun"
Re: How do you ground something in space? (Score:1, Informative)
From Wikipedia:
Circuit ground versus earth
In an electrical circuit operating at signal voltages (usually less than 50 V or so), a common return path that is the zero voltage reference level for the equipment or system.
Voltage is a differential quantity, which appears between two points having some electrical potentials. In order to deal only with a voltage (an electrical potential) of a single point, the second point has to be connected to a reference point (ground) having usually zero voltage.
This signal ground may or may not actually be connected to a power ground. A system where the system ground is not actually connected to earth is often referred to as a floating ground.
Re:OS? (Score:2, Informative)
First about the "all software is NASA controlled" assertion.. Well.. While I was watching NASA TV, I caught a glimpse of one of the astronauts obviously attempting to retrieve some e-mail from his laptop.. And then complaining over the com that he was getting a "you can only have one instance of Outlook running" - ground control advised for a laptop reboot, but the guy upstairs wasn't too keen on doing that (apparently, to him, this meant it was a server problem !) - the capcom person at that time then seemed to be taking the diplomatic side and answered : "lemme check" !
Second, you claim that NASA mathematically prove software correctness.. However, it is a known fact that this is an impossible thing to do (Rice's theorem)..
The fact that these system have no OS is debatable ! They at least need some system oriented code to interface between the hardware and the software (call it OS, library, firmware, whatever !) - but it seems that even getting the thing to initialize is failing..
Last, everyone is talking about the 'russian' computers.. Well, this guy last night in the press conference did state these were actually "western style" *european* computers !
--Ivan
Re:DFMEA (Score:3, Informative)
Hrm, the summary is different than the article; the article stated that "two of the six computers are running" which means 4 are down, not 2. Whichever is correct, any time more than one computer goes down, you have to look for common-cause failures.
Also, according to the article the US computers don't control attitude thrusters and that particular life support system, so the state of the US computers doesn't matter.
(Note for the anonymous poster above, but I didn't want to post twice: "common cause" means "the same situation makes multiple things break in the same way," not "that cause happens often".)
Re:OS? (Score:5, Informative)
The personal communication laptops the astronauts have are windows machines. The machines that run both ISS and Shuttle are **not**. They are derivatives of UNIX, and, as grandparent said, have many eyes and many thousands of dollars poured into each line of code. There was a good article not too long ago in Fast Company [fastcompany.com] about the shuttle coding team.
From the article: the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. That's impressive. The same care went into the ISS computers, at least from the US's side. I can't speak for Russia as I don't have that level of familiarity with them.
Last, everyone is talking about the 'russian' computers.. Well, this guy last night in the press conference did state these were actually "western style" *european* computers !
The Russian computers failed. The US computers have 'taken over' temporarily. Why? Because we have this nice little satellite network called TDRSS (Tracking and Data Relay Satellite System) which lets us relay communications with shuttle over the vast majority of the orbit. Russia does not. They can only communicate over line of sight, which is a few times each day for about 8 or so minutes.
Update: Computers down again (Score:4, Informative)
If disconnecting the power doesn't fix them problem, then the situation is even worse... the station will have to be evacuated next Wednesday, and would no longer have attitude control. It is likely that it would tumble out of control before any new mission could be made, making it impossible to dock the the ISS and probably resulting in its eventual re-entry.
Things are not looking good.
Re:Graphite failure (Score:5, Informative)
Re:NASA uses 30-year old UNIX derivative (Score:3, Informative)
VxWorks isn't a UNIX, it is a real time operating system from Wind River [windriver.com]. Its has POSIX compliance in a decent number of areas so writing a thread / task is similar to programming for UNIX, but it can be quite a different beast when it comes to actually running the software. My experience is that once you have the various application tasks debugged, it'll run practically forever. Though as the parent noted, a bad driver can spoil that in unexpected ways.
Re:you've conspiracy on the mind timmie (Score:3, Informative)
Re:OS? (Score:3, Informative)
Re:DFMEA (Score:3, Informative)
"guessing" at the problem and having "theories" is probably not a good way to go.
Welcome to the real world of problem solving. Any solution always starts out as a guess. It's pretty much impossible to solve any problem without eliminating a whole bunch of possibilities (i.e. guessing and having theories). It's likely 10 times harder when you don't have the tools necessary to diagnose this particular problem. (i.e. they need an oscilloscope to look for strange power fluctuations from the new solar array). So I could see how it might be particularly hard to turn those guesses and theories into near certainties.
Also, it's apparently a common-mode failure, which you shouldn't have in a safety-critical system; generally this is avoided by having different computer hardware and/or completely different code to do the same tasks.
Having completely redundant systems down to the electrical level is hard enough on the ground. In a small space station a few hundred miles in space I imagine it's next to impossible. You could argue that "why didn't they just have a $50 UPS that'd at least provide them with redundant power, then see if the broken computer boots". But then you have to realize this is a space station with limits on how much space there is for tools that might otherwise go un-used.
You could probably equally blame this issue on lack of testing. Though that's obviously difficult as well since you don't have a duplicate space station orbiting the earth to first try it out on.
Re:DFMEA (Score:4, Informative)
The US computers do however control the CMG's - the backups for the attitude thrusters[1], and the life support for the US side of the station. So even the loss of all the Russian computers wouldn't leave the station in trouble. (Unless CMG desaturation was required - which doesn't happen all that often.)
[1] Next year, IIRC, a second set of CMG's goes active and then the CMG's become primary with the attitude thrusters going into the backup role.
Re:Yakov Smirnoff says: (Score:1, Informative)
Re:Stopping rule (Score:3, Informative)
"One of the intentions was allegedy for it to be used for construction in space."
I doubt that intention was seriously advanced by engineers who know anything about it. To build something in space needs the parts (as pre-assembled on the ground as possible) and someone to bolt them together. A different big thing to build first isn't a great idea.
"Clearly there are advantages to being able to build spacecraft and such not intended for travel through atmosphere."
Next to the advantages of getting to do the building on the ground? You're going to build the biggest modules you can lift and design them to bolt together with as little work done in space as possible; just as they have with the ISS. Coming back from Mars in a vehicle without heat shielding and meeting a re-entry vehicle in earth orbit is a fine idea, but you don't need a station for that.
"For a mars mission especially, it would need to house what is basically a mission control center, as the radio delays to Houston would cause significant problems."
What?!? Radio delays between the ISS and Mars vs. the ground and Mars are not significantly different, nor even consistenly positive. On the scale of going to Mars, the ISS is at Earth. Heck the ISS would be behind the earth half the time, unable to broadcast to Mars at all.
"So why has that space construction capability all but been scrapped? "
There never was any such capability. The ISS is a construction PROJECT. The construction CAPABILITY is provided by the shuttle.
"Without that it is little more than a really expensive version of a normal space station."
A "normal" space station? The ISS is currently the *only* space station. Hard to get more normal than that. It sucks not because it's suckier than other space stations, but because space stations don't have a worthwhile role in our current space activities.