Forgot your password?
typodupeerror
Communications Space The Military United States Science Technology

Could a Dirty Rag Take Out a $2 Billion Satellite? 297

Posted by Soulskill
from the dumb-mistakes-cost-the-most dept.
An anonymous reader writes "The alleged rescue of a U.S. military communications satellite underscores some of the weaknesses in U.S. space efforts. Quoting: 'The seven-ton “AEHF-1,” part of a planned six-satellite constellation meant to support radio communication between far-flung U.S. military units, had been in orbit just one day when the problems began. The satellite started out in a highly-elliptical, temporary orbit. The plan was to use the spacecraft’s on-board engine to boost it to a permanent, geo-stationary orbit. But when the Air Force space operators at Los Angeles Air Force Base activated the engine, nothing happened. The Government Accountability Office would later blame the failure on a rag left inside a fuel line by a Lockheed worker.'"
This discussion has been archived. No new comments can be posted.

Could a Dirty Rag Take Out a $2 Billion Satellite?

Comments Filter:
  • by Anonymous Coward on Monday January 09, 2012 @03:14PM (#38640406)

    Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

    Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

    Shhh... I can dream!

    • by JonahsDad (1332091) on Monday January 09, 2012 @03:20PM (#38640492)
      TFA states that they are seeking compensation from Lockheed. Hopefully, that'll happen without an actual suit.
    • by Anonymous Coward on Monday January 09, 2012 @03:30PM (#38640640)

      Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

      Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

      Shhh... I can dream!

      Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

      • by Anonymous Coward on Monday January 09, 2012 @03:45PM (#38640814)

        Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

        Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

        Shhh... I can dream!

        Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

        Isn't that what should happen? I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever? If there is restitution for lost funds as well as improvements to try to prevent a repetition of the same problem, shouldn't everyone involved be satisfied? I'm fairly certain that the OP's hope that we all get some kind of tax refund is probably not going to happen, and even if it did, you'd be talking about a few dollars per person at most.

        • by Hatta (162192) on Monday January 09, 2012 @05:25PM (#38642314) Journal

          I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever?

          Forgetting a rag is an honest mistake. Failing to plan for honest mistakes by implementing the appropriate checks into your process is negligence.

          • by syousef (465911) on Monday January 09, 2012 @06:31PM (#38643356) Journal

            I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever?

            Forgetting a rag is an honest mistake. Failing to plan for honest mistakes by implementing the appropriate checks into your process is negligence.

            The engineer following the process is not necessarily the person that created the procedure. Also even if a procedure is in place double failures do occur - they are just less likely.

            I love the way so many people are willing to judge that a man should or should not be fired based on 3 minutes of reading a slashdot story. Really enhances my faith in human nature. Hope none of you ever sit on a jury. What disciplinary action if any should be faced by various staff involved is something that would require at least weeks of investigation, IF you want to go in that direction and waste the time on a witch hunt instead of just fixing the issue.

  • by fahrbot-bot (874524) on Monday January 09, 2012 @03:17PM (#38640444)

    blame the failure on a rag left inside a fuel line

    Must be a really small rag or really big fuel line. Seriously, how would this happen? It's a freaking satellite engine, not the shuttle main.

    • by localman57 (1340533) on Monday January 09, 2012 @03:19PM (#38640480)
      Yeah. Damn it people! This is just rocket science, not brain surgery!
    • by Richard_at_work (517087) <richardprice@nOSPam.gmail.com> on Monday January 09, 2012 @03:30PM (#38640652)

      You can use tiny squares of cloth, impregnated with cleaning solution, to clean the inside of valves and metal lines - gets rid of metal filings which are left over from the boring process.

      Quite easy to leave one behind. Which is why there are processes in place designed to prevent such issues.

      • by Maximum Prophet (716608) on Monday January 09, 2012 @03:57PM (#38640992)

        You can use tiny squares of cloth, impregnated with cleaning solution, to clean the inside of valves and metal lines - gets rid of metal filings which are left over from the boring process.

        Quite easy to leave one behind. Which is why there are processes in place designed to prevent such issues.

        So, they built a tool to make sure the rag was removed. Then they built another tool to check that the first tool was removed...

        More seriously, why wouldn't groundside testing notice that there was a rag in the line?

        • by Richard_at_work (517087) <richardprice@nOSPam.gmail.com> on Monday January 09, 2012 @04:15PM (#38641244)

          More seriously, why wouldn't groundside testing notice that there was a rag in the line?

          Yup, why wouldn't it?

          Obviously it didn't. Multiple times. In multiple different situations - this isn't the first space mission to be ruined because of something left where it shouldn't have been.

          The obvious answer to your question might be because it didn't block anything during testing, so there was the appearance of nothing wrong. Turn on the fuel flow, after the experience of the launch, and it might have been jostled free from wherever it had chosen to hide - from there it might be a short ride to a bottle necking point such as a crimp in the line, a sharp bend, or a valve, and thus begins the blockage.

        • Billions of dollars in technology, but nobody with an air nozzle hooked up to an air compressor found at any car mechanic's shop to blow out a fuel line.

        • by SwedishChef (69313) <craig&networkessentials,net> on Monday January 09, 2012 @04:35PM (#38641582) Homepage Journal

          More seriously, why wouldn't groundside testing notice that there was a rag in the line?

          Some of these positioning rockets are single-use. If you test one you have to build another to replace it. And then test it. And then.....

    • by pipingguy (566974)
      Junk left in piping can cause all kinds of problems (like, duh). That's why there are procedures to clean out the lines prior to startup. If the rag was left in oxygen or oxygen-rich piping, well...kaboom.

      One space industry insider, who spoke with The Diplomat on condition of anonymity, says lapses like the forgotten rag indicate a lack of experience in the lower ranks of U.S. space contractors. âoeIt was probably a mix all too common in the USAF programs: 80-year-old PhDs and 20-year-old college gra
  • by jandrese (485) <kensama@vt.edu> on Monday January 09, 2012 @03:18PM (#38640446) Homepage Journal
    Isn't this sort of like asking if a $5 wrench could wreck a car engine if it were left inside of a cylinder? Is anybody going to say "no"?

    And yes, I went with the car analogy right from the start. Deal with it.
    • by Suki I (1546431) on Monday January 09, 2012 @03:20PM (#38640494) Homepage Journal

      Exactly! Any old blockage could prevent fuel from getting through the fuel line. Same with the oxidizer. Even smashing a bug under an electrical component could cause a failure.

      • Re: (Score:3, Informative)

        by galaad2 (847861)

        bugs being smashed in electical components has already happened, lots of times in history.
        Here's one of the first properly documented cases of it, from 1947:

        http://www.history.navy.mil/photos/images/h96000/h96566k.jpg [navy.mil]

        Photo #: NH 96566-KN (Color)

        The First "Computer Bug"

        Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University, 9 September 1947. The operators affixed the moth to the computer log, with the entry: "First actual case of bug being found". They put out the word that they had "debugged" the machine, thus introducing the term "debugging a computer program".
        In 1988, the log, with the moth still taped by the entry, was in the Naval Surface Warfare Center Computer Museum at Dahlgren, Virginia.

        Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988.

        NHHC Collection

    • by xrayspx (13127) on Monday January 09, 2012 @03:25PM (#38640564) Homepage
      Luckily, in the aerospace industry, there's no such thing as a "$5 wrench". Hell that was probably a $700 dirty rag.
    • by shentino (1139071)

      Is it the same 5 dollar wrench that was used to beat the password out of the programmer of another sattelite that was hacked?

  • by apcullen (2504324) on Monday January 09, 2012 @03:18PM (#38640460)
    It's hard. Any little thing that goes wrong will likely cause the whole thing not to work.
    That's why it's rocket science.
  • Test Sequence? (Score:4, Interesting)

    by Anonymous Coward on Monday January 09, 2012 @03:20PM (#38640486)

    Who puts an engine together without a test fire? Seems to me that some simple checks would have prevented a very big waste of funds and effort. I guess it won't be a total waste if they can learn from it.

  • by Lumpy (12016) on Monday January 09, 2012 @03:20PM (#38640502) Homepage

    Assembly failure - leave a rag.
    Inspection failure - did not check for rag.
    Pre-flight final inspection - still did not find the rag.

    Wow, complete failure all the way down the line from assembly to mating with the launch vehicle.

    • by Massacrifice (249974) on Monday January 09, 2012 @03:24PM (#38640548)

      XXI century new space programs motto : It's failures all the way down, man!

    • by geek (5680) on Monday January 09, 2012 @03:26PM (#38640572) Homepage

      It reminds me of those surgery horror stories where the surgeon or staff leaves behind clamps and sponges inside the persons body.

      Shit happens. All we can really do is our very best to try and prevent it, but ultimately, we're human and prone to mistakes.

      • by roc97007 (608802) on Monday January 09, 2012 @03:45PM (#38640824) Journal

        > It reminds me of those surgery horror stories where the surgeon or staff leaves behind clamps and sponges inside the persons body.

        Funny you should mention that. I had emergency surgery last year for severe traumatic internal bleeding (won't bore you with the details -- or maybe I already have) and things happened so quickly that they did not have enough time for an instrument inventory. (Apparently it's someone's job to keep track of how many tools get used and then count them before final suture.) So after they got me stable they ran me back through x-ray to look for stuff. Didn't find anything, fortunately.

        But really -- it's not that much of a horror story, they just have to open you back up at some point to retrieve the objects. It's not something you want to have happen, but it's a fairly well known procedure. Horror stories to me are things like taking off the wrong limb [1] or prescribing catastrophically wrong medication.

        [1] Before I went in for knee surgery, the doctor gave me a sharpie and had me mark the correct knee. Just in case.

    • by Pope (17780)

      It does make me curious as to how big the rag and fuel line are. Also makes a great juxtaposition with the story immediately below on the home page, The Challenges Of Building A Mars Base [slashdot.org]!

    • by Overzeetop (214511) on Monday January 09, 2012 @03:29PM (#38640628) Journal

      Actually pre-flight final won't catch that kind of thing; it's already buried in the system (and you don't fire thrusters on a flight unit prior to launch). This is likely one of those cases where a scrap of cleaning"rag" was torn off within the path in an area not visible at either end and went unnoticed. To save money, a visual of the system prior to final assembly was determined to be sufficient and the endoscope procedure was eliminated, saving several thousand dollars (combined on all the lines). Sure, in hindsight a compressed air test would have been sufficient, but it's a little late to play what-if now.

      • Sure, in hindsight a compressed air test would have been sufficient, but it's a little late to play what-if now.

        Except that devising a simple $1000 test might save the next $2,000,000,000 satellite. Extra points if you can add to or replace an existing test that tests multiple systems sufficiently.

        • The test was devised, mostly likely. There are hundreds - no, probably tens of thousands - of tests on this craft. You can probably trace every single raw material element back to where it was mined, refined, billeted, shipped, stored, manufactured, machined, binned, tagged, selected, gaged, installed, torqued, tested, and approved for flight. Every single time a human touches a part it costs $100 (well, that was a decade ago, it's probably $200 now).

          The anal retentiveness of the work flow on a satellite is

    • by LWATCDR (28044) on Monday January 09, 2012 @03:37PM (#38640736) Homepage Journal

      You left out Slashdot summary failure.

      FTFA
      "On Oct. 24, AEHF-1 reached its originally planned orbit. Testing began soon afterward. The Air Force expects to bring the satellite into service in March. Meanwhile, two more AEHFs are slated to launch in 2012."

      They got it into the correct orbit over two months ago using the small thrusters.
      In other words...
      More sensationalistic headlines to get clicks and comments from the new Slashdot.
      Really? Oh and the answer is "no a dirty rag did not take out a 2 billion dollar commsat."
       

      • by Baloroth (2370816)
        And of course that "underscores some of the weaknesses in U.S. space efforts." Actually, I would say it underscores the strength: they managed to fix the problem using ingenuity and scarce resources. Also, a "scrap of cloth" != "a rag". Calling it a rag implies someone just forgot a whole piece of cloth. A scrap of cloth implies it ripped or was otherwise accidentally and through no negligence (well, not gross negligence anyways, they may still have checked more carefully) deposited.
  • At least one of the recent Soyuz failures was put down to a similar issue - debris left in a fuel line by a worker.

    • In soviet russia, rag washes out you!

  • by JoeMerchant (803320) on Monday January 09, 2012 @03:23PM (#38640526) Homepage

    So, somebody can't come up with the used rag disposal accounting paperwork and the GAO concludes that it must have been left inside?

    I mean, this kind of thing is good for sponges during surgery, why not satellite assembly?

  • by rts008 (812749) on Monday January 09, 2012 @03:23PM (#38640532) Journal

    So, the problem is the satellite is 'on the rag'?

  • by pla (258480) on Monday January 09, 2012 @03:27PM (#38640586) Journal
    FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine).

    So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.
    • by hrvatska (790627)

      FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine). So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.

      The GAO was probably basing its conclusion on statements from Lockheed itself. According to this [bloomberg.com] it was Lockheed that concluded the problem was some cleaning material left in the line.

      "It should not have happened,” Deputy Under Secretary of the Air Force for Space Programs Richard McKinney said. “It was a quality mistake and we took steps to make sure it does not happen again,” he said. “It was obviously a very serious error.”

      “It appears that there was a blockage in one of the fuel lines,” McKinney said. Lockheed thinks “it was caused by some cleaning material that was used in a line that was not properly vacated when they went through production.”

  • Heading hyperbole (Score:5, Informative)

    by biometrizilla (1999728) on Monday January 09, 2012 @03:28PM (#38640608)
    Already been established that they were able to overcome the rag and get the satellite into a functional orbit where it can fulfill its mission objective. http://www.spaceflightnow.com/atlas/av019/120103rescue.html [spaceflightnow.com]
  • by wisebabo (638845) on Monday January 09, 2012 @03:28PM (#38640620) Journal

    "Finally, it speaks to the size and age of the U.S. space arsenal that the Air Force felt it had no choice but to rescue AEHF-1 instead of replace it with a back-up spacecraft. 'The asset inventory is getting so tight that they spent months limping the heap to its proper orbit,' the insider lamented."

    Look guys, before you throw away (replace with a backup) a $2 Billion satellite, I damn well hope you try some pretty heroic measures. Those are my tax dollars in (the wrong) orbit! So I'm very glad you didn't have (to use) a backup satellite.

    Anyway, does anyone know if the low power thrusters which were eventually used to put this satellite into the correct orbit used the same fuel tank as the clogged thruster? Otherwise 1) I'm very surprised they had enough fuel to get there and 2) they would probably have very little left to last the lifetime of the mission. So let's hope that all the thrusters used a central (hydrazine?) fuel tank and there's plenty left.

    Space is hard and while the U.S. program has certainly had its ups and downs at least it hasn't seen the near total collapse as what happened to the Ruskies. They had quite a bad year last year and that blogger walking around their factory just exposed their problems more. If Mars is going to be a "Red" planet it will because of China not Russia.

    • by Fallon (33975)
      Yes, it used the same fuel source. There were some efficiency issues, but it did not alter the expected lifespan of the satellite on orbit.
    • by LoveMuscle (42428)

      According to this:

      http://spaceflightnow.com/atlas/av019/111009.html [spaceflightnow.com]

      They used the hall effect thrusters instead of the hydrazine/nitrogen tetraoxide engine. The hall effect thrusters run on xenon and electricity, so NO they did not use the same fuel source. The hall effect thrusters have a specific impulse of ~8000s instead of the ~300s for hydrazine, so they are insanely fuel efficient, but extremely low thrust. (1/4N vs ~450N for the main engine)..

      • by LoveMuscle (42428)

        Ok.. It appears that they used both. I should have read the whole article.. The hydrazine was expended in a 5N thruster over 12 firings (raising the satelite about 1/7 of the way and changing the inclination), then the 0.25N hall effect thrusters were used for the remaning 19400miles (firing 12hours a day for 8 months)..

  • RAG?? (Score:2, Informative)

    by Anonymous Coward

    A Dirty Rag? C'mon - RTFA! "The blockage 'was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,' according to the Government Accountability Office."

    That could mean a tiny fragment of fabric. It's not like they put a rag in the gas tank to keep gas from leaking out. sheesh.

  • One info I have yet to see in any of the stories I have read on this.

    The "main" engine doesn't start so they use thousands of firings of the maneuvering thrusters to circularize the orbit. Do the "main" & maneuvering thrusters use the same fuel source or has the mission longevity been compromised? Does anyone know?

    • by Fallon (33975)
      No, there were some efficiency issues, but it did not compromise it's expected on-orbit lifespan.
  • It's not like they have little nanites with cameras crawling around there. Fine, the main thrusters aren't working, but how did they manage to specifically blame it on a piece of rag in a fuel line? Aren't there a lot of ways a thruster can fail to fire?
  • "Finally, it speaks to the size and age of the U.S. space arsenal that the Air Force felt it had no choice but to rescue AEHF-1 instead of replace it with a back-up spacecraft. &ldquo;The asset inventory is getting so tight that they spent months limping the heap to its proper orbit,&rdquo; the insider lamented. "

    Translation:

    We spent tens, perhaps hundreds of thousands, heck maybe even a couple million on labor to save a $2,000,000,000 dollar satellite rather than build another 2 billion dollar sate
  • by Ambitwistor (1041236) on Monday January 09, 2012 @04:00PM (#38641018)

    We must be increasingly on the alert to prevent our enemies from taking over our satellite fuel lines, thus knocking out our military communications. Mr. President, we must not allow a dirty rag gap!

  • by ThatsNotPudding (1045640) on Monday January 09, 2012 @04:07PM (#38641140)
    just like in invasive surgery, there would be a known count of 'sponges' and after buttoning up, they had all better be accounted for.
  • by j00r0m4nc3r (959816) on Monday January 09, 2012 @04:29PM (#38641500)
    Why do they dismiss the possibility of alien parasites that look like rags and feed on satellite engine fuel?
  • by satuon (1822492) on Monday January 09, 2012 @04:48PM (#38641782)

    Reminds me about all those stories of bottles put inside cars during assembly. Here's a funny one (albeit fictional):

    A man goes to a car dealership one day after inheriting a good deal of money (or after a great business deal, whatever -- he has a lot of money somehow). After looking around the lot, he picks out the nicest, newest, fanciest, most expensive car he can. He pays cash up front and drives out of the dealership in the new car.

    On his way home, he starts hearing a rattling sound -- something must be wrong. So he turns around and goes right back to the dealer. The dealer is of course very sorry, and offers to either fix the car or let the man take a different one while they order a replacement. The man really wants the car, so he just has the guy fix it. Two hours later, the mechanics give the car back, saying they couldn't find a thing wrong with it. The man is a bit wary, but he drives home. Whatever the rattle is, it has stopped.

    A day or so later, the rattle starts again. He takes it to the dealership, and they still can't find anything wrong with it. This continues for a number of weeks -- sometimes the rattle even goes away on its own. Anyway, after nearly two months of it, the dealer is very upset -- he doesn't want to get a bad reputation. So he orders a replacement and exchanges it with the man for the malfunctional car.

    Then he orders the mechanics in the shop to do a complete tear-down to figure out the problem. They begin taking the car apart, piece by piece, but they can't find anything -- until they take apart the door. Inside, they find a piece of metal pipe, along with a note. Written on the note, in a scrawling, worker's hand is: "So, you finally found the rattle, you rich son-of-a-bitch."

Often statistics are used as a drunken man uses lampposts -- for support rather than illumination.

Working...