Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Space

Upside-Down Sensors Caused Proton-M Rocket Crash 323

Posted by Unknown Lamer
from the someone's-getting-fired dept.
Michi writes "According to Anatoly Zak, the crash of the Russion Proton rocket on 1 July was apparently caused by several angular velocity sensors having been installed upside down. From the source: 'Each of those sensors had an arrow that was supposed to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.' It seems amazing that something as fundamental as this was not caught during quality control. Even more amazing is that the design of the sensors permits them to be installed in the wrong orientation in the first place. Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection) could have prevented the accident." A review of the quality control procedures used by the contractors responsible is underway.
This discussion has been archived. No new comments can be posted.

Upside-Down Sensors Caused Proton-M Rocket Crash

Comments Filter:
  • ...aren't so amazing when you look at the track record of Russian manufacturing.

    • Re: (Score:3, Funny)

      by Anonymous Coward

      Quality Control in Russia basically consists of hitting it with a mallet, and if it doesn't fall apart on impact, it passes.

    • by PetiePooo (606423) on Wednesday July 10, 2013 @10:19AM (#44238897)

      ...aren't so amazing when you look at the track record of Russian manufacturing.

      Before we Americans point too many fingers, let's not forget NASA is not immune to similar mistakes. [wikipedia.org]

      • My favorite part of that one was the crash was investigated by the MIB - Mishap Invetigation Board and definitely not Men in Black.
      • Probably outsourced to WalMart, anyhow. Even a communist space agency has a budget, and why not use OTS spare parts from one of the American Government's largest suppliers?

        http://www.theonion.com/articles/walmart-wants-republican-president,15517/ [theonion.com] http://www.theonion.com/articles/dhs-teams-up-with-walmart,18722/ [theonion.com]

        OR ( inclusive or) : over-educated engineers assumed the arrows were the spin state of the subatomic detectors inside. A quasi-random distribution of Up and Down would be required to det

        • by lgw (121541)

          It's just Murphy's Law in action. The original inspiration for Murphy's Law was when Murphy flew a test airplane that had instrument gauges that, just like the sensors in TFA, could physically be installed upside down, but it was completely obvious which way was right. All of the gauges in his test plane were upside down, leading him to coin the phrase that has far outlasted our memory of his career as a test pilot.

          Murphy's Law was specifically about installing aerospace instrumentation upside down.

      • things are always unstable during test periods. once a device this complicated, like a space booster or a 787 for instance, gets certified and enters serial production, that is the part where inspections and workers empowered to shut down the line becomes the paramount safety mechanism.

      • don't forget this one - the standard / metric measurement confusion which caused the crash of the NASA Mars orbiter http://www.cnn.com/TECH/space/9909/30/mars.metric/ [cnn.com]
  • by alen (225700) on Wednesday July 10, 2013 @10:01AM (#44238589)

    being from there i bet half the people working on this came to work drunk and/or hung over most days

    • The Party anti-drinking agitprop [retronaut.com] didn't really seem to make a dent during the Soviet years.

  • by Anonymous Coward on Wednesday July 10, 2013 @10:02AM (#44238599)

    Murphy's Law is still in effect. Like the snippet says make sure that they can only be installed one way mechanically, because you won't catch 100% of the errors in QA.

    • by X0563511 (793323)

      What stops the key from being installed wrongly?

      • Theoretically, any geometric irregularity. Take SIM cards or SD Cards, for example. Put a notch somewhere and bang, you can't mount it in any other position.

        • by oobayly (1056050) on Wednesday July 10, 2013 @10:11AM (#44238759)

          Never underestimate the ingenuity that people are capable of in order to install something wrong. Somebody in our office forced (yes, forced) a Xerox Phaser ink block in the the slot the wrong way round. The thing is basically a shape sorter that a toddler is capable of understanding.

          • Re: (Score:2, Funny)

            by Anonymous Coward

            there is a job waiting for them in the space optics division of Perkin-Elmer

          • by jrumney (197329) on Wednesday July 10, 2013 @10:18AM (#44238875) Homepage

            The greatest pleasure my toddler ever got from his shape sorter was when he discovered that the 3 could be forced through the hole for the C. Never underestimate the satisfaction a disgruntled office worker gets from jamming the ink block into the printer the wrong way around.

            • by KiloByte (825081) on Wednesday July 10, 2013 @10:29AM (#44239077)

              An old joke:

              A militia (communist police) station has been ordered to conduct an intelligence test. It consisted of a board with three holes: a circle, a triangle and a square, and three corresponding blocks. The next days, the commandant announces: I'm very proud of our station: all of you passed the test! 5% have shown exceptional intelligence, 95% exceptional strength!

            • by pspahn (1175617)

              When I was about 12, I took a knife and shaved off the corners of a molex connector and attached it an old hard disk that had tons of bad sectors. I just wanted to see what would happen.

              While interesting to a 12 year old, it was nothing more than a couple pops, some smoke, and a little bit of melted plastic. It was at this very moment that I learned everything I ever needed to know about computers.

          • by LordLimecat (1103839) on Wednesday July 10, 2013 @11:05AM (#44239633)

            Ive seen RAM modules installed backwards. "Wait!", you say, "Isnt there a notch which prevents that?" Well, yes, there WAS a notch...

          • by Dishevel (1105119)

            I love my 8560MFP.

          • by aralin (107264)

            Reminds me of when local police introduced a shape sorter as interview screening tool,100% of the applicants passed, 50% with brains, 50% with force.

        • by X0563511 (793323)

          That works if it's a solid item that you can mold like that - but if it were sheets of metal, you could bolt it on wrongly etc. Granted it would certainly reduce the chances...

      • What stops the key from being installed wrongly?

        The design of the key and the tooling and processes used to produce it. Speaking generally you use behavior shaping constraints [wikipedia.org] which prevent incorrect assembly. Proper design, interlocks, jigs and fixtures, automated tooling, and lots of other tools are used to eliminate mistakes.

        Anything that relies on visual inspection by a human WILL eventually have an error. My company makes wire harnesses and every time we are forced to rely on a visual inspection process there inevitably are some errors. Most of

    • by jeffmeden (135043) on Wednesday July 10, 2013 @10:05AM (#44238655) Homepage Journal

      What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

      Obligatory: It's not rocket science!

      • by Athanasius (306480) <slashdot@mig[ ]org ['gy.' in gap]> on Wednesday July 10, 2013 @10:26AM (#44239015) Homepage

        My reading of 'angular velocity sensor' is that they're meant to sense rotation. If you're sat stationary on the pad there is no such rotation and thus you'll get a 'correct' zero reading. You'd have to perform such a test during some known movements of the rocket (part).

      • by Baloroth (2370816) on Wednesday July 10, 2013 @10:27AM (#44239041)

        What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

        Obligatory: It's not rocket science!

        The sensors in question were for angular velocity. Given that pre-launch the craft doesn't have any (peculiar) angular velocity, the sensors would return the correct results (zero) no matter how they were installed.

      • What seems more amazing is that a simple software check pre-launch (i.e. "do all the sensors think they are pointed up?") was not part of the SOP. Given that their exact function is orientation detection, skipping the opportunity for self-test via that function is somewhat baffling.

        No - the sensors were 'angular velocity sensors'. They do not measure orientation but change of orientation. Is a bit more difficult to check pre-launch than an orientation sensor.

        • by putaro (235078)

          The Proton arrives at the pad horizontally and is then erected into a vertical position for launch. If the electronics are powered up at that point, you could run an angular velocity check/

      • by Deadstick (535032)

        Given that their exact function is orientation detection

        Except that it's no such thing. An angular velocity sensor senses, well, angular velocity. That means speed of rotation. A broken clock is right twice a day, and a stationary angular velocity sensor is right all the time.

        • by oobayly (1056050)

          and a stationary angular velocity sensor is right all the time.

          Not if it's giving a non-zero value at rest

          <ducks>

      • by CastrTroy (595695)
        They are angular velocity sensors. At launch, their angular velocity is 0. Although there probably should be a check pre-launch to determine if all the sensors are indeed facing the correct way, or better yet, make it impossible to install them upside down.
      • by MiG82au (2594721)
        Which part of "angular velocity" do you not understand?
    • by TheCarp (96830) <sjc AT carpanet DOT net> on Wednesday July 10, 2013 @10:19AM (#44238889) Homepage

      Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law [wikipedia.org]

      Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

      The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

      So this is potentially, very much related to the original usage.

      • by smellsofbikes (890263) on Wednesday July 10, 2013 @12:52PM (#44241373) Journal

        Amusingly, when someone actually attempted to track down who murphy was, and where the law came from.... http://en.wikipedia.org/wiki/Murphys_law [wikipedia.org]

        Edward Murphy proposed using electronic strain gauges attached to the restraining clamps of Stapp's harness to measure the force exerted on them by his rapid deceleration. Murphy was engaged in supporting similar research using high speed centrifuges to generate g-forces. Murphy's assistant wired the harness, and a trial was run using a chimpanzee.

        The sensors provided a zero reading; however, it became apparent that they had been installed incorrectly, with each sensor wired backwards. It was at this point that a disgusted Murphy made his pronouncement

        So this is potentially, very much related to the original usage.

        If I remember right, the way a wheatstone strain gauge is set up, there are four ways to connect it. One is right, two are wrong but give you half the resolution you expected -- so you get data, just lousy data -- and one is completely wrong and you get no data whatsoever. It was hooked up in the completely wrong configuration. That was what made him so mad: there was only a 25% chance it would get hooked up in the completely worthless configuration, but that's what happened.

    • by Bartles (1198017)
      The mechanical interlock is not the solution here. Two letters on or next to the arrow is. An indicating arrow without a prescribed orientation is useless. Cardboard box manufacturers know this. UP.
      • by Lumpy (12016)

        Yes it is. even as simple as using a 3mm screw on one end and a 5mm screw on the other end. attempting to install it wrong will instantly cause a problem because one screw will not go through the hole.

        Problem is, most places are hiring lowest wage workers, so you do not get people that have the IQ to understand that if the screw does not fit then something is wrong, they just get more 3mm screws or more likely the even more stupid foreman or manager tells them to.

        Same goes if you use a keying hole and p

    • Even the simplest of mechanical interlocks (such as a notch at one end that must be matched with a corresponding projection)

      This only moves the problem, it doesn't fix it. There is now the possibility for the sensors to be installed correctly into mechanical interlocks that were themselves installed upside down

    • by PPH (736903)

      can only be installed one way mechanically,

      Won't help. The assembly notes say, "Beat to fit. Paint to match."

  • by Oloryn (3236) on Wednesday July 10, 2013 @10:02AM (#44238615)
    Wasn't something like this responsible for the formulation of Murphy's law?
  • Hey, give them a break! I do that in Kerbal Space Program all the time!
    • by oobayly (1056050)

      Squad must be ecstatic, KSP has become the de-facto analogy when it comes to space related tutorials pretty much everywhere.

      • When the rocket guides itself into the ground because you put something on upside down, it's the most appropriate analogy. I've done exactly that more than once, to the point where my mental pre-launch checklist includes "navball completely blue?"
    • by Rhacman (1528815)
      The difference between reality and Kerbal Space Program is that in KSP you test your hugely expensive designs by just launching them and seeing if they crash whereas in reality... what point was I trying to make again?

      (Anyway, they should have just let Jeb pilot that thing. It might have still fireballed, but he's pulled off far stranger feats!)
  • by jovius (974690) on Wednesday July 10, 2013 @10:10AM (#44238735)

    Should have launched from Australia.

  • by Ellis D. Tripp (755736) on Wednesday July 10, 2013 @10:18AM (#44238865) Homepage

    which plowed into the desert floor without deploying any parachutes because a G-switch was installed backwards...

    http://www.universetoday.com/73/genesis-accident-report-released/ [universetoday.com]

  • by dkleinsc (563838) on Wednesday July 10, 2013 @10:24AM (#44238979) Homepage

    "Whoopth, I had the thilly thing in reverthe!"

  • by 140Mandak262Jamuna (970587) on Wednesday July 10, 2013 @10:28AM (#44239047) Journal
    Back in the day when I was with the ministry of defense we lost a vehicle due to an error like this. They had changed the vendor for the gyros of the roll sensor. The new gyros had the voltages in the reverse sense. It is possible one vendor was European and the other was American. They wired it according to the sense of the old vendor. So the control input to the ailerons would add to the roll instead of counteracting it. The RPV crashed 1.5 seconds after launch.

    In the postmortem the flight director started with, "... we sadly lost the vehicle after a flight of 1.5 seconds ...". The mission director interrupted, "What flight? The damned thing had a 6000 Kg[sic][*] rocket booster. You can put it under a 3 ton rock and it will 'fly' for more than 2 seconds..."

    [*]He should have said 6000 Kgf-sec, because that was the impulse delivered by the twin rocket boosters each 1500 Kgf thrust burning for 2 seconds.

  • We wouldn't want anything to 'appen to it...

    .
  • by Okian Warrior (537106) on Wednesday July 10, 2013 @10:41AM (#44239259) Homepage Journal

    The US once sent a probe all the way to mars, only to have it fail because the ground computer was in imperial units while the orbiter was in SI units [wikipedia.org].

    Getting everything correct is hard... really hard. For most projects you have elaborate "fail gracefully" modes which rely on external agents to notice the problem and take action. A doctor or pilot can take appropriate action, but it's hard to do with rockets.

    For comparison, I wrote the software for the altimeter that goes into some 747 aircraft. Total of about 21,000 lines of C, about 40% comments so figure 12,000 lines of code. The testers (and I) worked really hard to find all bugs in the system, knowing that a mistake could knock a plane out of the sky. There were elaborate internal checks both in software and process, and Boeing did their own testing on top of ours. Everything passed, all requirements were met, things looked good.

    The device had 1 bug, found after installation. A software typo which wasn't caught by QA even though it had a specific testing requirement. No one was negligent, it just slipped by despite best efforts.

    Multiply this by all the devices in an aircraft, and add in the other engineering disciplines like electronics and mechanical. It's really hard to get everything right all at once, and on the first try.

    • by Lumpy (12016)

      But did you test it with absurd conditions? I rarely see any software testing that they say," ok let's install all the sensors backwards and see what it does" They assume that never happens and never test for what is assumed as "impossible to ever happen".

      • The short answer is "yes".

        All functions range-checked their arguments on entry, calculations range-checked their results before performing further calculations, precondition logic was tested to ensure the preconditions held, periodic testing checked as many "things that should never happen" as we could think of.

        We never ignored a possibility because it was absurd, so long as there was a way to test it it was tested. The difficulty is coming up with a comprehensive list of things to check... very hard to do

  • Quick! Before it's too late! Somebody call the Australian Space Agency!
    Tell them to look for any boxes not marked: \/ Fragile: Then End Down \/

  • by Antipater (2053064) on Wednesday July 10, 2013 @10:45AM (#44239319)

    I'm confused by this explanation. An upside-down angular velocity sensor would definitely pitch the rocket out of control the way it did. But what about the brown plume that was clearly visible before the rocket lost it? The consensus seemed to be that that was unburned rocket fuel, implying an engine shutdown.

    I don't build rockets, but I can't see how an upside-down rotation sensor could cause an engine shutdown, especially since the shutdown occurred before the rocket began pitching.. Could there have been more than one problem on the rocket?

    • by Lumpy (12016)

      The computer could certainly shut down a rocket given bad information. IT probably wigged completely out and simply shut down that engine.

    • I am not sure about the Proton, but earlier Russian boosters used differential throttling of the engines to control the attitude. The brown is probably excess oxidizer from running it off-mixture to throttle the engine and control the attitude - in this case to chase the erroneous gyro readings.

  • heh (Score:5, Funny)

    by wbr1 (2538558) on Wednesday July 10, 2013 @10:53AM (#44239441)
    In Soviet Russia, snesors installed correctly, rocket installed upside down.
  • Interesting article on Space News
    http://www.spacenews.com/article/launch-report/36112proton-launch-failures-more-likely-when-russia-footing-the-bill#.Ud2DnPkyZ8E/ [spacenews.com]
    that points out that the Proton launch failures have a mysterious correlation to whether the customer is private or government (with government launches being the unlucky ones).
  • If you want to go to space.

    If it starts pointing toward space you are having a bad problem and you will not go to space today.

  • by drwho (4190)

    I am confused - did the upside-down sensors cause the other problems as well, such as the early disconnect of wiring, or are these all separate failures? If it's the latter, there needs to be some serious effort made to improve the design and construction.

  • We just assume that the sensors were upside down -- but does anyone ask if the rocker wasn't upside down and the sensors right side up?

    No. No they do not. Installing sensors is a thankless job and nobody says; "Great sensor." They only talk to you if something goes wrong."

    >> Brought to you by the Anti Sensor Installer Defamation League

    • by petes_PoV (912422)
      All this talk implies that the sensors were installed in a vehicle with a pointy end pointing upwards. Isn't it entirely possible that the sensors were installed at some point well before the vehicle was commissioned and there was no "up" or "down" (only left or right) on the parts that the sensors get attached to. Maybe the sensors _were_ installed correctly, but the whole subassembly was the wrong way round?
  • by bill_mcgonigle (4333) * on Wednesday July 10, 2013 @11:06AM (#44239663) Homepage Journal

    Can the flight control system verify the sensor readings before launch? "Sensor 7 says the rocket is pointing towards the Earth on the launchpad - we might want to have a look".

  • by 50000BTU_barbecue (588132) on Wednesday July 10, 2013 @11:12AM (#44239751) Homepage Journal
    Should they have launched from Australia instead?

    Yeah, weak joke, sorry.

  • When you try to make things cheaper you get failures.

  • The Thor missile, the first of the intermediate range ballistic missiles had the same issue on its 2nd launch causing the range operator to hit the self destruct button in fear that it was going the wrong way.
  • Contractors? (Score:4, Interesting)

    by whitroth (9367) <whitroth&5-cent,us> on Wednesday July 10, 2013 @11:46AM (#44240309) Homepage

    The Russians are using contractors, now?

    On the other hand, they seem to be doing vastly better than the US these days - we have NO WAY to put someone in orbit (unless the Pentagon's got a black program).

    We also had Challenger and Columbia. And on the latter note, I'll add that I believe my late ex's analysis, rather than the "it's falling insulation" answer. She was an engineer, and worked at the Cape for 17 years, including on the Shuttle, and she thought that some of the inspections that were supposed to be done were *not* being done, or not being done as frequently as they were supposed to have been... and that the hydraulic lines broke due to stress corrosion microcracking, and there went the aerilons.

    So, how many astronauts/cosmonauts have the Russians lost lately?

                      mark

  • by LoRdTAW (99712) on Wednesday July 10, 2013 @02:27PM (#44242545)

    You wouldn't need a notch, you simply move the screw holes around so they aren't square. The best method is a trapezoid pattern. Two screw holes are set closer together. Impossible to mount upside down or sideways. Or simply shift one screw hole like the ATX power supplies do.

  • by Chas (5144) on Wednesday July 10, 2013 @03:23PM (#44243189) Homepage Journal

    And the K19.
    And the K141 (The Kursk)
    Soyuz 1
    Soyuz 11
    And about half a dozen other fatal accidents involving shoddy workmanship.

  • by PPH (736903) on Wednesday July 10, 2013 @03:48PM (#44243499)

    ... did the same thing with a seven segment timer on a bomb. He thought he had 50 minutes until it started counting seconds down "6h, 8h, Lh ...". Then Bender turned it over.

    You'd think the Russians would study other industries lessons learned and best practices.

"Hey Ivan, check your six." -- Sidewinder missile jacket patch, showing a Sidewinder driving up the tail of a Russian Su-27

Working...