Upside-Down Sensors Caused Proton-M Rocket Crash 323
Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident."
A review of the quality control procedures used by the contractors responsible is underway.
The quality conrol problems... (Score:2, Insightful)
...aren't so amazing when you look at the track record of Russian manufacturing.
Re: (Score:3, Funny)
Quality Control in Russia basically consists of hitting it with a mallet, and if it doesn't fall apart on impact, it passes.
Re:The quality conrol problems... (Score:5, Informative)
...aren't so amazing when you look at the track record of Russian manufacturing.
Before we Americans point too many fingers, let's not forget NASA is not immune to similar mistakes. [wikipedia.org]
Re: (Score:2)
Re: (Score:2)
http://www.theonion.com/articles/walmart-wants-republican-president,15517/ [theonion.com] http://www.theonion.com/articles/dhs-teams-up-with-walmart,18722/ [theonion.com]
OR ( inclusive or) : over-educated engineers assumed the arrows were the spin state of the subatomic detectors inside. A quasi-random distribution of Up and Down would be required to det
Re: (Score:3)
It's just Murphy's Law in action. The original inspiration for Murphy's Law was when Murphy flew a test airplane that had instrument gauges that, just like the sensors in TFA, could physically be installed upside down, but it was completely obvious which way was right. All of the gauges in his test plane were upside down, leading him to coin the phrase that has far outlasted our memory of his career as a test pilot.
Murphy's Law was specifically about installing aerospace instrumentation upside down.
you mean the part where "ours always blow up?" (Score:2)
things are always unstable during test periods. once a device this complicated, like a space booster or a 787 for instance, gets certified and enters serial production, that is the part where inspections and workers empowered to shut down the line becomes the paramount safety mechanism.
Re: (Score:2)
Re: (Score:2)
Actually, the Russians can make quality products, provided they are paid what they are owed and on time.
I bet there was some "extra payment" that was not paid... and whoops, you know things happen...
That may be, but consider this - do you want to
It won't be a lump of coal he will be bringing.
Re: (Score:3, Insightful)
There's a difference between true communism [wikipedia.org] and corrupt, dictatorial regimes.
For those too lazy to click on the link:
A perfect example of true communism applied to a specific field would be open-s
Re:The quality conrol problems... (Score:4, Insightful)
Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present. Related entries: No True Scotsman; Ivory Tower Intellectualism.
Re: (Score:2)
Re:The quality conrol problems... (Score:5, Interesting)
Actually, that's wrong. Capitalism is designed with corruption and greed in mind. Greed motivates entities to perform better in the market and get more stuff. Corruption is dealt with by entities shifting to competitors who are screwing them less. Capitalism's failure is in assuming all involved entities are sufficiently intelligent to be aware of when they're being screwed, and principled enough to forgo what they want to avoid being screwed.
Re: (Score:2, Informative)
Communism(n) [also commernussum, comianizzem] - Anything that we don't do in the USA, such as believing that the Earth is more than 6,000 years old.
Re:The quality conrol problems... (Score:5, Insightful)
Communism (n) - an unattainable standard that is constantly held up as a model of perfection despite having no functional real world example past or present.
Sorry, but that's nonsense. All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).
Stalin made a pretty good attempt, but didn't quite succeed.
Re:The quality conrol problems... (Score:4, Insightful)
All you need to do to create perfect communism is kill everyone else so no-one can disagree with you (you can't just kill the ones who disagree, because the others might only be pretending to agree).
Stalin made a pretty good attempt, but didn't quite succeed.
By an incredible coincidence, that's also the way to create a perfectly free market with no government intervention.
Re:The quality conrol problems... (Score:4, Insightful)
Democracy /capitalism "work" even if you dont have a "pure" implementation.
Communism has never worked in any of its forms; its just gotten millions killed in purges and famines, and left nations in a crippled, dysfuncitonal state even decades later. The cry has always been that it hasnt worked because it wasnt implemented in a pure enough form, hence my "no true scotsman" comment.
Capitalist / democratic states, however, continue to be represented by every major world power. China is becoming a major power precisely by embracing a functioning economic system that looks and smells an awful lot like capitalism.
Re: (Score:3)
A perfect example of true communism applied to a specific field would be open-source software.
You sure about that? You can't apply communism "to a specific field". That would be like saying that disaster relief volunteers are an example of communism. Also, I'm pretty sure that my means of software production, namely my brain, is not subject to common ownership. Unless somebody's borrowing it in secret whenever I'm asleep, that is.
Re:The quality conrol problems... (Score:5, Insightful)
>That would be like saying that disaster relief volunteers are an example of communism
Actually that sounds about right to me. "From each according to their ability, to each according to their need" and all that.
Most family households also run on at least partially communist principles. Luxuries may need to be at least partially earned (or not, plenty of douches with entitlement issues out there), but it's a sad family where everyone's *needs* aren't taken care of first.
The problem with communism seems to be that it doesn't seem to scale well beyond the tribe/monastery/commune level. Once the population gets too large to allow for effective communal decision making, communal ownership tends to become de-facto ownership by the decision makers, massively exacerbating the problem of corruption.
Re:The quality conrol problems... (Score:4, Insightful)
It looked good on Paper doesn't it.
However it rarely works for a long time or with a lot of people.
The problem is that we live in a world of scarcity, we can't get all that stuff that we want, or need. The communist system tries to make everything equal, however that means everyone will be living in scarcity, and not really having what they need or want. Because everyone will be wanting, it will open the door for someone(s) to cheat the system and try to get more, once they have more they will hold on to it. And the system begins to fail.
Software like Open Source tends to work better, because there isn't a limit in supply. You can copy share make a copy of the copy and continue on and on. There is no scarcity in the Open Source Model.
Re: (Score:3)
This is your typical right-wing FUD right there. You don't understa
Re: (Score:3)
That's BS.
There's a Titanium smelting plant in Redditch. Granted, it's Russian owned, but still, it's in the UK.
vodka and work don't mix (Score:5, Interesting)
being from there i bet half the people working on this came to work drunk and/or hung over most days
Re: (Score:2)
The Party anti-drinking agitprop [retronaut.com] didn't really seem to make a dent during the Soviet years.
Re: (Score:3)
Russians are quite capable of making some really sturdy stuff. If anything, they engineer it to stand up to operators who are often uneducated and half-drunk. That is where you get masterpieces like the AK-47 which isn't the most accurate assault rifle out there, but you can bury one in shit for a year, dig it up, clean out the mechanism a bit and it will still fire. Try that with an M-16 and you'll have a military-themed door stop.
So, I would not be surprised if there was someone working on that Proton-
QA is not the problem (Score:3, Insightful)
Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.
Re: (Score:2)
What stops the key from being installed wrongly?
Re: (Score:2)
Theoretically, any geometric irregularity. Take SIM cards or SD Cards, for example. Put a notch somewhere and bang, you can't mount it in any other position.
Re:QA is not the problem (Score:4, Interesting)
Never underestimate the ingenuity that people are capable of in order to install something wrong. Somebody in our office forced (yes, forced) a Xerox Phaser ink block in the the slot the wrong way round. The thing is basically a shape sorter that a toddler is capable of understanding.
Re: (Score:2, Funny)
there is a job waiting for them in the space optics division of Perkin-Elmer
Re:QA is not the problem (Score:5, Insightful)
The greatest pleasure my toddler ever got from his shape sorter was when he discovered that the 3 could be forced through the hole for the C. Never underestimate the satisfaction a disgruntled office worker gets from jamming the ink block into the printer the wrong way around.
Re:QA is not the problem (Score:5, Interesting)
An old joke:
A militia (communist police) station has been ordered to conduct an intelligence test. It consisted of a board with three holes: a circle, a triangle and a square, and three corresponding blocks. The next days, the commandant announces: I'm very proud of our station: all of you passed the test! 5% have shown exceptional intelligence, 95% exceptional strength!
Re: (Score:3)
When I was about 12, I took a knife and shaved off the corners of a molex connector and attached it an old hard disk that had tons of bad sectors. I just wanted to see what would happen.
While interesting to a 12 year old, it was nothing more than a couple pops, some smoke, and a little bit of melted plastic. It was at this very moment that I learned everything I ever needed to know about computers.
Re:QA is not the problem (Score:5, Funny)
Ive seen RAM modules installed backwards. "Wait!", you say, "Isnt there a notch which prevents that?" Well, yes, there WAS a notch...
Re: (Score:2)
I love my 8560MFP.
Re: (Score:2)
Reminds me of when local police introduced a shape sorter as interview screening tool,100% of the applicants passed, 50% with brains, 50% with force.
Re: (Score:2)
That works if it's a solid item that you can mold like that - but if it were sheets of metal, you could bolt it on wrongly etc. Granted it would certainly reduce the chances...
Behavior shaping constraints (Score:2)
What stops the key from being installed wrongly?
The design of the key and the tooling and processes used to produce it. Speaking generally you use behavior shaping constraints [wikipedia.org] which prevent incorrect assembly. Proper design, interlocks, jigs and fixtures, automated tooling, and lots of other tools are used to eliminate mistakes.
Anything that relies on visual inspection by a human WILL eventually have an error. My company makes wire harnesses and every time we are forced to rely on a visual inspection process there inevitably are some errors. Most of
Re:QA is not the problem (Score:5, Insightful)
What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.
Obligatory: It's not rocket science!
Re:QA is not the problem (Score:5, Insightful)
My reading of 'angular velocity sensor' is that they're meant to sense rotation. If you're sat stationary on the pad there is no such rotation and thus you'll get a 'correct' zero reading. You'd have to perform such a test during some known movements of the rocket (part).
Re: (Score:2)
Re: (Score:2)
Re:QA is not the problem (Score:5, Informative)
What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.
Obligatory: It's not rocket science!
The sensors in question were for angular velocity. Given that pre-launch the craft doesn't have any (peculiar) angular velocity, the sensors would return the correct results (zero) no matter how they were installed.
Re: (Score:3)
Sure, if this was a vertically mounted accelerometer. It was not: it was an angular velocity sensor (I believe this [upz.ru] is the technical specification page, although I could be wrong). The angular velocity of the craft on the launchpad is zero. If it's not, you have much much bigger problems than an improperly mounted sensor.
Re:QA is not the problem (Score:4, Interesting)
IT IS NOT ZERO, its 0.004187 degrees per second around some vector. The Earth is turning!
Re: (Score:3)
Not correct. If you cannot accurately sense this rate (~15 deg/hr) you will go far off the trajectory in a 10-15 minute boost. It absolutely cannot tolerate errors of a few degrees. Checking for the proper rates is an absolutely standard pre-launch check and typically, any biases in this are calibrated out while on the pad based on the known alignment of the rocket and the lat/long of the pad.
Re: (Score:2)
What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.
No - the sensors were 'angular velocity sensors'. They do not measure orientation but change of orientation. Is a bit more difficult to check pre-launch than an orientation sensor.
Re: (Score:2)
The Proton arrives at the pad horizontally and is then erected into a vertical position for launch. If the electronics are powered up at that point, you could run an angular velocity check/
Re: (Score:2)
Given that their exact function is orientation detection
Except that it's no such thing. An angular velocity sensor senses, well, angular velocity. That means speed of rotation. A broken clock is right twice a day, and a stationary angular velocity sensor is right all the time.
Re: (Score:2)
and a stationary angular velocity sensor is right all the time.
Not if it's giving a non-zero value at rest
<ducks>
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
You move the rocket in a known way, as in lift it, then read the outputs.
Re: (Score:2)
Gravity.
Since these things are really accellerometers,either the sensor or the computer needs to ajust for local gravity to give accurate velocity measurements.
In this case, the reading should have been around -9.8m/s2 * 2 = -19.6m/s2 when the rocket was sitting on the pad.
Re: (Score:2)
Spin the part they are installed in, before the rocket is assembled. Sure you have to spin one sub assembly, but at least not the whole thing.
Re:QA is not the problem (Score:5, Informative)
Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law [wikipedia.org]
So this is potentially, very much related to the original usage.
Re:QA is not the problem (Score:4, Informative)
Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law [wikipedia.org]
So this is potentially, very much related to the original usage.
If I remember right, the way a wheatstone strain gauge is set up, there are four ways to connect it. One is right, two are wrong but give you half the resolution you expected -- so you get data, just lousy data -- and one is completely wrong and you get no data whatsoever. It was hooked up in the completely wrong configuration. That was what made him so mad: there was only a 25% chance it would get hooked up in the completely worthless configuration, but that's what happened.
Re: (Score:2)
Re: (Score:2)
Yes it is. even as simple as using a 3mm screw on one end and a 5mm screw on the other end. attempting to install it wrong will instantly cause a problem because one screw will not go through the hole.
Problem is, most places are hiring lowest wage workers, so you do not get people that have the IQ to understand that if the screw does not fit then something is wrong, they just get more 3mm screws or more likely the even more stupid foreman or manager tells them to.
Same goes if you use a keying hole and p
Redesign is not the solution (Score:2)
Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection)
This only moves the problem, it doesn't fix it. There is now the possibility for the sensors to be installed correctly into mechanical interlocks that were themselves installed upside down
Re: (Score:3)
can only be installed one way mechanically,
Won't help. The assembly notes say, "Beat to fit. Paint to match."
(The Real) Murphy's Law strikes again! (Score:3)
Re:(The Real) Murphy's Law strikes again! (Score:5, Interesting)
Yes, the real, original Murphy's law apparently came from Col. Stapp, who was testing rocket sleds for the rocket program.
I should note that the putative original Murphy's Law reads, "If there are two or more ways to do something, and one of those ways can result in a catastrophe, then someone will do it." [murphyslaws.net]. The website goes on to say "This is a principle of defensive design, cited here because it is usually given in mutant forms less descriptive of the challenges of design for lusers. For example, you don't make a two-pin plug symmetrical and then label it `THIS WAY UP'; if it matters which way it is plugged in, then you make the design asymmetrical."
Highly appropriate to the topic, I might say. If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.
For those who wish to nit-pick my attention to detail and editing, also, I will for further irony include the wikipedia link, as well: http://wikipedia.org/wiki/Muphry%27s_law [wikipedia.org]
Re:(The Real) Murphy's Law strikes again! (Score:5, Informative)
Re: (Score:2)
I'd give you a point if I had any to give.. I fell for that hook, line, and sinker!
Re:(The Real) Murphy's Law strikes again! (Score:5, Funny)
If only they had labeled, with the arrow, the words "up", and put another arrow down, with the letters "dn" for "down", then none of this would have happened.
Except that "dn" upside-down is indistinguishable from "up". Murphy strikes again?
KSP (Score:2)
Re: (Score:2)
Squad must be ecstatic, KSP has become the de-facto analogy when it comes to space related tutorials pretty much everywhere.
Re: (Score:2)
Re: (Score:2)
(Anyway, they should have just let Jeb pilot that thing. It might have still fireballed, but he's pulled off far stranger feats!)
Wrong hemisphere (Score:5, Funny)
Should have launched from Australia.
Re:Wrong hemisphere (Score:4, Funny)
That's the problem. Some parts were made by subcontractors in Australia, where up is down, rats are as tall as humans and hop around and flat-chested 30-years-old women are classified as underage teenagers.
Shades of the US "Genesis" sample return probe... (Score:5, Interesting)
which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...
http://www.universetoday.com/73/genesis-accident-report-released/ [universetoday.com]
Duck Dodgers in the 24 1/2th Century (Score:4, Funny)
"Whoopth, I had the thilly thing in reverthe!"
I have seen something similar. (Score:4, Interesting)
In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."
[*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.
Nice cosmodrome you've got here, Colonel... (Score:2)
.
Mars orbital failure (Score:5, Insightful)
The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units [wikipedia.org].
Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.
For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.
The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.
Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.
Re: (Score:2)
But did you test it with absurd conditions? I rarely see any software testing that they say," ok let's install all the sensors backwards and see what it does" They assume that never happens and never test for what is assumed as "impossible to ever happen".
The short answer is "yes" (Score:3)
The short answer is "yes".
All functions range-checked their arguments on entry, calculations range-checked their results before performing further calculations, precondition logic was tested to ensure the preconditions held, periodic testing checked as many "things that should never happen" as we could think of.
We never ignored a possibility because it was absurd, so long as there was a way to test it it was tested. The difficulty is coming up with a comprehensive list of things to check... very hard to do
Wait, Upside down you say? (Score:2)
Quick! Before it's too late! Somebody call the Australian Space Agency!
Tell them to look for any boxes not marked: \/ Fragile: Then End Down \/
What about the brown plume? (Score:3)
I'm confused by this explanation. An upside-down angular velocity sensor would definitely pitch the rocket out of control the way it did. But what about the brown plume that was clearly visible before the rocket lost it? The consensus seemed to be that that was unburned rocket fuel, implying an engine shutdown.
I don't build rockets, but I can't see how an upside-down rotation sensor could cause an engine shutdown, especially since the shutdown occurred before the rocket began pitching.. Could there have been more than one problem on the rocket?
Re: (Score:2)
The computer could certainly shut down a rocket given bad information. IT probably wigged completely out and simply shut down that engine.
Re: (Score:3)
I am not sure about the Proton, but earlier Russian boosters used differential throttling of the engines to control the attitude. The brown is probably excess oxidizer from running it off-mixture to throttle the engine and control the attitude - in this case to chase the erroneous gyro readings.
heh (Score:5, Funny)
Somebody got it in for the Russian government? (Score:2)
http://www.spacenews.com/article/launch-report/36112proton-launch-failures-more-likely-when-russia-footing-the-bill#.Ud2DnPkyZ8E/ [spacenews.com]
that points out that the Proton launch failures have a mysterious correlation to whether the customer is private or government (with government launches being the unlucky ones).
This end should point toward the ground (Score:3)
If you want to go to space.
If it starts pointing toward space you are having a bad problem and you will not go to space today.
causality (Score:2)
I am confused - did the upside-down sensors cause the other problems as well, such as the early disconnect of wiring, or are these all separate failures? If it's the latter, there needs to be some serious effort made to improve the design and construction.
sure, blame the sensor guy. (Score:2)
We just assume that the sensors were upside down -- but does anyone ask if the rocker wasn't upside down and the sensors right side up?
No. No they do not. Installing sensors is a thankless job and nobody says; "Great sensor." They only talk to you if something goes wrong."
>> Brought to you by the Anti Sensor Installer Defamation League
Re: (Score:2)
Sofware Fix? (Score:3)
Can the flight control system verify the sensor readings before launch? "Sensor 7 says the rocket is pointing towards the Earth on the launchpad - we might want to have a look".
Just a question (Score:3)
Yeah, weak joke, sorry.
Cost cutting (Score:2)
When you try to make things cheaper you get failures.
US did the same thing (Score:2)
Contractors? (Score:4, Interesting)
The Russians are using contractors, now?
On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).
We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.
So, how many astronauts/cosmonauts have the Russians lost lately?
mark
Simple fix (Score:3)
You wouldn't need a notch, you simply move the screw holes around so they aren't square. The best method is a trapezoid pattern. Two screw holes are set closer together. Impossible to mount upside down or sideways. Or simply shift one screw hole like the ATX power supplies do.
From the same people that brought you Chernobyl... (Score:3, Informative)
And the K19.
And the K141 (The Kursk)
Soyuz 1
Soyuz 11
And about half a dozen other fatal accidents involving shoddy workmanship.
Professor Farnsworth ... (Score:3)
You'd think the Russians would study other industries lessons learned and best practices.
Re:In Soviet Russia (Score:5, Funny)
Re: (Score:2, Offtopic)