NASA Finds Critical Assembly Fault in Shuttle 383
mzs writes "During corrosion inspection on Discovery, technicians noticed that one of the gears in a rudder actuator had been installed backwards. This particular actuator was the top-most of four that control the air brakes on the tail. As luck turns out, if it had been the bottom-most actuator, loss of the shuttle and crew would have been nearly inevitable. Plans are in place to have four spares by the time Shuttle missions resume next year."
Simple solution, really. (Score:5, Funny)
Could they not stamp "THIS SIDE UP" or whatever on the components?
Re:Simple solution, really. (Score:5, Insightful)
Re:Simple solution, really. (Score:5, Funny)
Re:Simple solution, really. (Score:5, Funny)
Re:Simple solution, really. (Score:2)
Re:Simple solution, really. (Score:5, Funny)
Could they not stamp "THIS SIDE UP" or whatever on the components?
They did something similar with the modified 747 that carries the shuttle orbiter back after landing. See this picture [purdue.edu]. This is supposed to be a sign on top of the 747, where the orbiter links to the top of the 747's fuselage. It reads "Place Orbiter Here...Black Side Down".
If this is real, they have one hell of a sense of humor.
Re:Simple solution, really. (Score:3, Funny)
'Guaranteed delivery anywhere in world in 30 minutes'
'- or the next one's free'
I also liked the story on slashdot a few years ago about a decommisioned secret RSA listening station with a huge smiley face painted on the parabolic radio receiver dish saying 'hi' to the russians...
Re:Simple solution, really. (Score:3, Funny)
On my SSBN we had two signs over the firing console. The first said "Trident - When you care enough to send the very best", the other said "16 empty missile tubes, 16 mushroom clouds, It's Miller Time". Yes, strategic weaponeers have a black sense of humor.
Re:Simple solution, really. (Score:5, Interesting)
FRONT TOWARDS ENEMY.
Here is a pic [bobzook.com]
Re:Simple solution, really. (Score:5, Funny)
Quick...it's total darkness, the bad guys are on your trail and you have 15 seconds to place and rig the claymore, or you're all gonna get captured or killed.
Yes you want it to be as completely foolproof as possible.
Re:Simple solution, really. (Score:3, Insightful)
Colour code the connections.
From the article:
'The gear fits into the assembly both ways, but is slightly asymmetric so the teeth do not fit exactly if the gear is reversed.'
So why not have the side its supposed to go in green, and the side where its not supposed to go in red?
Simple visual solution that can be spotted quicker?
NeoThermic
Re:Simple solution, really. (Score:4, Insightful)
Re:Simple solution, really. (Score:3, Funny)
Mine doe!@*&934128()#*!)@((~!
Re:Simple solution, really. (Score:2)
Re:Simple solution, really. (Score:4, Funny)
Re:Simple solution, really. (Score:5, Funny)
There is no "up" in space....
Re:Simple solution, really. (Score:2)
Re:Simple solution, really. (Score:5, Interesting)
In fact the original inspiration for Murphy's Law was a G-force meter that was installed backwards, thereby taking meaningless readings. (It probably didn't go below zero.)
Another example is the 1969 gearbox fire on the Canadian navy ship, HMCS Kootenay. A gearbox bearing was installed backwards, which restricted its flow of lubrication oil (on a naval vessel, the gearbox is the size of a car and absorbs tremendous loads). Apparently it did say which way to install it, but the installation was made in a foreign shipyard where the workers could not read English. The poorly lubricated bearing overheated and caused an explosion during a full-power trial; nine sailors were killed and dozens injured.
The moral of all these stories is: if it's important which way something is installed, make it asymmetric so that it's physically impossible to install it the wrong way. Labels are not enough.
Re:Simple solution, really. (Score:5, Insightful)
I believe that the NTSB found a stamping on a rail car wheel caused a failure and derailment, but i'll be danged if I can find the incident. Maybe tonight...
Moisture could collect in the little crevices of the letters leading to corrosion. i suspect that this is not really a concern.
Silk screening or some creative powdercoating could totally avoid these issues. But what do I know. I managed to smoke my e-machine 500 last night. Nobody would hire me to be a shuttle engineer!
Re:Simple solution, really. (Score:5, Funny)
Except they stamped the wrong side (Score:3, Insightful)
Great idea, but the workers stamped the wrong side on some versions, and the part cannot be remade in time.
This is a real problem in industry, you can put any sign on something, but then you gotta make sure the signs are right too. Indeed the wrong sign leads some workers to put it in backwards, even knowing the right way, while others will get in the habbit of putting it in with the lettering wrong, and not correct themselves when the next version is ships with the right parts.
Re:Simple solution, really. (Score:4, Interesting)
However, the manufacturer stamped the wrong side of the pivot, and it was dutifully installed upside-down. Murphy wins again no matter what you do.
How can a fault go unnoticed for so long? (Score:5, Interesting)
I thought they even checked Airplanes more thoroughly
Re:How can a fault go unnoticed for so long? (Score:2, Insightful)
Re:How can a fault go unnoticed for so long? (Score:3, Insightful)
Re:How can a fault go unnoticed for so long? (Score:4, Funny)
Careful, jerking your knee so suddenly like that might result in an injury.
Re:How can a fault go unnoticed for so long? (Score:5, Insightful)
As for horribly flawed: Compared to whom? Spaceflight is dangerous. Minor oversights that in most industries would cause a misprint in a news article tend to blow up and kill people.
Basically, until you land something off-planet, you have no room to talk. If you want to point to someone who is doing a better job of it, you pretty much have the Russians for comparison, and they have had even more problems.
-Ryan C.
Re:How can a fault go unnoticed for so long? (Score:4, Informative)
I think the reality is more like "Money bullsh*ts".
Re:How can a fault go unnoticed for so long? (Score:3, Insightful)
Yeah I know I know... but damn people you have to draw the line somewhere. I wish all things worked that well impropperly installed.
Re:How can a fault go unnoticed for so long? (Score:4, Funny)
Re:How can a fault go unnoticed for so long? (Score:2)
Instead of slamming NASA (Score:4, Interesting)
It'd be nice to give some credit for the people that have put in layer upon layer upon layer of safeguards to check for exactly this sort of thing and the dilligent people that find this stuff. And caught it.
The awful thing is that this is going to be just another reason for Congress to loot the NASA money bag.
Re:Instead of slamming NASA (Score:5, Informative)
Maybe you missed the details, but this has been in place on the Discovery for over 20 years.
-molo
Re:Instead of slamming NASA (Score:2)
Yes, indeed. Their performance has been exemplary. However in the future, finding problems somewhat faster than 20 years after construction might be a good thing.
Re:Instead of slamming NASA (Score:2, Offtopic)
Not NASAs Fault (Score:2)
The sub should be punished for their lack of effort and watched closer in ongoing efforts.
Re:Instead of slamming NASA (Score:2)
Re:Instead of slamming NASA (Score:3, Interesting)
Re:Instead of slamming NASA (Score:5, Insightful)
This might make sense if Congress (or any group of politicians) was actually interested in SAVING money anywhere. No tinfoil hat is needed to know that anytime a politician takes money away from one line item in the budget, they are merely diverting it to another that involves their own interests.
Re:Instead of slamming NASA (Score:3, Interesting)
I'd say that, while that does seem quite inconsequential, at one point we thought that the flow of molasses couldn't be all that crucial [mv.com] either, until the lives of 21 people depended upon it.
Transparency. (Score:5, Interesting)
It was, however, just a matter of time before a Columbia-type disaster occured. The suttle program has a remarkable safety record, Challenger and Columbia no matter.
Re:Transparency. (Score:2)
Why didn't they do this after Challenger? Are they so thick they needed two disasters in order to get serious about safety?
I have
Safety is relative (Score:2)
Get real. At ANY point in technology, reaching orbit is a heck of a lot harder than getting from point A to point B over paved roads in a car. One gallon of gas, properly vaporized, is roughly equivalent to a stick of dynamite. So most of us strap ourselves in with 10-20 sticks of dynamite, daily. The Shuttle has millions of poinds of fuel and oxi
Re:Safety is relative (Score:4, Insightful)
Why does the modern Soyuz have a better safety record than the Shuttle? Why did our old ballistic missiles have a better safety record than the Shuttle? Even the enormous Saturn V rockets never had an accident in flight.[1]
Why does the Shuttle have such a terrible safety record relative to other rockets that attain orbit?
I'll tell you why: because it was over-ambitious. Congress was sold on the idea of a re-usable (read: cheap) launch vehicle that can do cool stuff like repair satellites. The truth of the matter is that if we had stuck with traditional launch vehicles (fire-once rockets), the money we saved over the long run would have allowed us to just replace failing satellites rather than repair them. (How many satellites have we repaired anyway?) We could even have built the space station for less. (Look at how we launched Skylab. Surely we could have repeated that a few times to get as large a space station as we wanted.) The legacy of the Shuttle is that of an overpriced, underperforming safety hazard.
All manned spaceflight is dangerous. The Shuttle is just more dangerous that most.
[1] The Apollo capsule had two serious accidents, one on the ground and one on the way to the moon.
Re:Safety is relative (Score:3, Insightful)
Re:Transparency. (Score:3, Insightful)
Re:Transparency. (Score:2)
That's nice.
they could have at least launched a rescue to get the astronauts off of the shuttle and onto another vehicle.
No they couldn't. There wasn't another vehicle available.
Re:Transparency. (Score:3, Informative)
If NASA managers had realized early on that Columbia had suffered a catastrophic breach in its left wing during launch - either by obtaining satellite imagery or, more likely, by having the astronauts stage an inspection spacewalk - they might have had time to mount a repair spacewalk or even an emergency rescue mission with the shuttle Atlantis, the chairman of the Columbia Accident Investigation Board said today. link [spaceflightnow.com].
Re:Transparency. (Score:3, Insightful)
Actually there was. The next Shuttle mission was being prepped for a launch a month later. If the situation had been observed and understood, Columbia could have gone into a reduced consumption mode to stretch out time on orbit, and preparations for the next launch could have been stepped up (skip the payload, etc).
The management failure was that they didn't even look.
I guess they... (Score:3, Funny)
Geez... (Score:4, Funny)
Re:Geez... (Score:5, Interesting)
"I'm kind of at the top of my field [in gaming]," he said. "When I started reading about aerospace, I realized there was an incredible level of things to learn.
Re:Geez... (Score:4, Interesting)
He blames it on the ten thousand different manufacturers you deal with in IT, ranging from motherboard suppliers, to RAM makers, to CPU makers, hard drives, UPSs, and of course, software. The pieces work, it's getting them to work together that's a bitch. With a satellite, you have maybe 20 or 30 people who, in combination, know everything about it and who can coordinate with each other.
Think about that. Keeping a mid-size server farm up and running smoothly (all the while undergoing constant upgrades, new feature additions, etc.) is more difficult than designing and launching a satellite. Straight from the horse's mouth.
Re:Geez... (Score:5, Informative)
I'm qualified to work on things like airframes and engines, and I can calculate a pretty mean orbit, if I do say so myself. But I'm lost when it comes to things like avionics or heat shield design. So "rocket science" is indeed complex and tricky, and a successful rocket design will require experts from many fields. But things like compressible flow, which seems to be what Carmack's talking about, aren't really outside the grasp of a dedicated student at all. And of course, all of this sounds like black magic to the nontechnical layman.
Of course, we don't go around telling people this, or we wouldn't be able to look down our noses at everyone else. "I design jet engines, and I've done some work on the Mars program. Oh, you write computer games? Aw, that's cute."
-Carolyn
So the solution to the shuttle's aging problem (Score:2)
Yep, thats a good plan, soon they'd be flying a bird full of gizzards and no egg. How about some consistant QC and management instead?
Re:So the solution to the shuttle's aging problem (Score:2)
according to the lore, it had been taped together so many times and had so many redundant systems it could fly with 95% of them defective... but of course it's just lore
Why are heads not rolling? (Score:3, Interesting)
---
http://thewired.blogs.com/teotwawki
The techno-mediated cultural conspiracy
Re:Why are heads not rolling? (Score:2, Insightful)
Re:Why are heads not rolling? (Score:5, Insightful)
The Shuttle is one of the (if not *the*) most complex bits of engineering we, as a world-wide society, have achieved. I would never expect any shuttle to be manufactured/assembled/flown/controlled/maintaine
Re:Why are heads not rolling? (Score:3, Insightful)
30 succesful flights is not "a job well done" in this case. If the actuators had been installed on the other side a catastrophic failure would have been not only possible, but probable.
What that is called is "dumb luck".
It's sort of like winning at slots in a casino and being told that is "a job well done".
I'm betting... (Score:5, Funny)
Outsourced Inspectors? (Score:2, Funny)
Capsules (Score:2, Insightful)
Re:Capsules (Score:2)
Do they even have reusuable capsules?
Besides, when have the rudders ever "jammed"?
A quote I remember .... (Score:4, Interesting)
"When the most intelligent work on the most complex to build the the only prototype, inevitably the radio won't work."
The point is that when working on very complex designs and prototypes installing something incorrectly doesn't seem odd because your brain is unable to "see" the mistake for what it is. In a car, if you install the brakes incorrectly, the scale is such that you understand the mistake simply from your gut, visually. Like looking at a crumpled front fender and understanding that's not correct.
time for a new one (Score:4, Interesting)
Should have used Java (Score:5, Funny)
I know NASA is conservative with technology, but using assembly in this day and age is way backwards!
Re:Should have used Java (Score:2)
Re:Should have used Java (Score:4, Insightful)
Bad Joke
Actually Assembly is a great language if your worried about efficiency and/or space. A lot of controllers in aerospace are coded that way and there is good reason for that. You can never get away from machine language for everything. Many times it is the best route to code in.
Just goes to show.. (Score:5, Interesting)
This stuff happens all the time (Score:5, Interesting)
An example: a while back, we had a test engine spewing fuel all over the test cell for no readily apparent reason, prompting a panic that an entire compartment of the engine would have to be redesigned from scratch--until one of the test engineers found a fuel line seal that had not been reinstalled in the engine after the last teardown and reassembly. How do you miss something like this when there's a careful set of instructions to follow for every step of the assembly? I don't know, but I do know that humans are fallible, so we are constantly dealing with a stream of lost, damaged, and defective parts. Anyway, they put the seal back in, and the engine worked fine. (I have an NDA, so this is not what actually happened, but it is analogous.)
When I was in school, the more I learned about the environment the shuttle operates in, the more I was impressed by the fact that it worked at all, and now that I'm learning more about manufacturing engineering (not what I studied for; stupid job market), I'm surprised that the shuttles have as few problems as they do.
-Carolyn
Twice?! (Score:5, Interesting)
Given the complexity of a system like the shuttle, it is not surprising that out of 1000s of components there could be a mistake in one of them (and given some redundancy and robustness, it is not surprising that the shuttle could fly 30 times with one or more poorly installed components, though one would not normally want to bet on that...).
However, two errors out of 8 actuators checked implies some serious quality control issues.
-Marcus
Re:Twice?! (Score:2)
It will be amusing if we discover later that the parts were put on backwards deliberately, to compensate for some engineering flaw. Well, not so amusing for the astronauts falling out of the sky on fire.
Re:Twice?! (Score:3, Insightful)
=Smidge=
When is a problem a problem? (Score:4, Insightful)
When does a defect become a problem? I wonder if this was really a Critical problem because shouldn't some indication have already been seen by now?
I mean since they have fixed this problem will two other problems surface that are more critical and maybe they should have left it alone?
Quothe NASA: (Score:2, Funny)
Murphy's Law (Score:3, Informative)
NASA Finds Critical Assembly Fault in Shuttle (Score:3, Funny)
Murphy's Law (Score:2)
As Edward A. Murphy, jr. said it: [geocities.com]
Nothing like seeing a faithful replication of the impetus for Murphy's observation...
Re:Murphy's Law (Score:4, Insightful)
The important lesson here is about the design of critical parts. Nothing important should be made asymetrical and reversable. Even labeling it "THIS SIDE UP" on one side and "WRONG! DANGER! WRONG!" on the other isn't good enough. The part should either be symetrical, so it doesn't matter which way it goes, or non-reversable, so it wont fit upside down/backwards. Important thing to remember in mechanical engineering (gears|levers|*) or even eletrical parts specifications (connectors should be keyed to mate only one way: the right way).
Outsourcing (Score:3, Funny)
Aren't there supposed to be 5 actuators? (Score:2, Interesting)
I always thought there were 5 of everything to keep surfaces working even after a double failure. With only 4 actuators, if 2 fail, and start working against the other 2, the working pair can't overpower the non-working pair and the surface is useless. With 5 actuators, it takes a triple failure before the surface won't work.
Slightly asymmetric? (Score:3, Interesting)
The mistake dates back to the actuator's assembly at Hamilton Sundstrand in Rockford, Illinois, and is not easy to spot. The gear fits into the assembly both ways, but is slightly asymmetric so the teeth do not fit exactly if the gear is reversed.
Show me a man who can find a slightly asymmetric shape, and I'll show you a man who can find a slightly tritriangular number.
Or a slightly odd one ... hey wait, that's me. Except I am not a number, I am a free man!
Yes, slightly assymetric... (Score:3, Informative)
Ever see a winch? The teeth on the gear there is an extreme of that sort - only designed to pull load in. So it's not done to be mean - it's probably done as to fit the spec.
Kjella
Engineering practices (Score:4, Insightful)
It's strange and somewhat disconcerting that this was not the case for this shuttle component, but I haven't seen the part in question.
New backwards moniker for NASA! (Score:3, Funny)
This whole thing is despicable.
This happens: See the F-111 program (Score:4, Interesting)
Re:This happens: See the F-111 program (Score:3, Insightful)
ANY system where the right side up/down of a single fucking nut put on by one single guy that does not have any kind of independent quality control/inspection/etc process, any system at all that allows a bolt put on backwards by one single guy to kill 6 people...
It's a systemic f
Same Story - Different Article (Score:4, Informative)
Stuff happens; learn from it. (Score:5, Interesting)
Things will go wrong. Learning how to cope when the evil wind blows is critical. In this case, we now know that the thing can be flown with one actuator in upside down. If the bottom one malfs, swap it out in orbit with the top one, and you still might get home. People are going to get killed doing this. People got killed learning to sail the Mediterranean. It's still worth doing.
manual (Score:5, Interesting)
I can imagine the guy that noticed this first. Probably went something like: looks at actuators. looks at diagram of how they're to be installed. looks at diagram again. looks at actuators. turns diagram around; notices that the legend is now upside down, so concludes that can't be it. checks other pages of diagram to see if this page is unusual--different view, maybe. finds that it isn't. checks back for errata. finds none.
Looks around. "Hey Bob, what do you make of this?" Thinks about all the work that day that isn't going to get done, because now management and, if he's lucky, congressional inspectors are going to crawl up his ass. At least he knows that he didn't *install* the things.
Fail Safe (Score:2, Interesting)
The ONLY way to guarantee that... (Score:2, Funny)
If the USA is to become the empire it plans to be, we need more draconian measures to keep the incompitentes out of the way. Something like this would suffice:
From: Microsoft Corporate
Subject: Trusted Computing Initiative 2
In an effort to better secure computing for our customers, we are implementing new measures in our code revision system. The
Maybe (Score:3, Funny)
Nasa Haters... (Score:4, Interesting)
On a side-note, the reason Nasa is stuck in the proverbial hard-place between multi-billion dollar budgets and missions that nobody cares about is that we've all started over-valuing human-life. It's ridiculous that space exploration all but stopped because of the 2 shuttle disasters. Certainly, the loss of those crews was tragic, but the best way to honor those crews is to relentlessly pursue the dream that they died for, not hamstring ourselves being overly cautious.
Call me old-fashioned, but I still believe there are things more important than one or a dozen human lives. IMO, exploring the universe is one of them.
the most complex engineered system ever (Score:4, Informative)
It's not suprising that there are flaws in the system - disasters lying dormant until the moment when they cause the destruction of the entire system.
This is one of the biggest arguements for a Vertical Takoff / Vertical Landing vehicle - it simplifies the system because it eliminates specialized components for landing.
Here's the mantra: fault tolerant systems. Things will fail. Can your space shuttle deal with those failures gracefully?
Re:the most complex engineered system ever (Score:3, Interesting)
Hubble flaw also due to a mis-installed part? (Score:3, Informative)
Re:Four Spare of what? (Score:2, Flamebait)
Re:I didn't read the article, but... (Score:3, Interesting)
On high-load gears, the teeth are sometimes designed so that the faces which mesh are perpendicular to the force they apply. This keeps the gears from pushing each other away when they are loaded, and makes the gears engage more positively. But as a consequence, the teeth cannot be perfectly symmetrical.
If one of the gears is installed upside down, then the teeth would be loaded on a smaller surface area than designed (since their faces are now not par