Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Space Bug NASA

Incorrectly Built SLS Welding Machine To Be Rebuilt 150

schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.

ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.

Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."

It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
This discussion has been archived. No new comments can be posted.

Incorrectly Built SLS Welding Machine To Be Rebuilt

Comments Filter:
  • is a penny burned

  • by Anonymous Coward

    The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.

    • I'm pretty sure that letting the contractors bilk NASA is the point of the exercise at this stage. The SLS isn't referred to as the Senate Launch System for nothing.
      • by Z00L00K ( 682162 ) on Saturday April 18, 2015 @12:56AM (#49498861) Homepage

        Since I'm working in a large organization I have come to realize that the amount of documentation in many projects is huge - often so large that essential key information is masked away, or right out FUBARed.

        It's also not uncommon that the customer requirements are "interpreted" by people with no technical knowledge whatsoever and they have a tendency to eradicate information that they think is "too technical", or information that they think drives unnecessary cost. Some people also have a tendency to rename things to a semantic that is to common people fuzzy. Even obfuscation occurs. At the same time documents are filled with a large number of pages listing old or discarded alternatives.

        • by frisket ( 149522 )
          I wonder if they bother using the (XML-based) systems that provide (enforce, even) the multiple views and accurate categorizations that were designed for serious heavy-duty tech doc. Or if they just stuff everything into Word. Maybe someone in the know can tell us.
        • I suppose too many TL;DR situations where it really counted. Though, I suspect, it would indicate that the documents were probably more to cover ass than communicate in a clear and concise fashion the requirements. Then again, some people are good at writing and poor at communicating. Case in point: I was once given a few paragraphs describing logic requirements for a new functionality in an application, though since I need to convert it into code, I first converted it into a logic flow in English and sent

        • Whenever the blame is on "miscommunications", the real cause is likely poor project management, both on the owner and contractor/subcontractor side. The owner needs a good PM in place to ensure their critical requirements are being met. The constructor PM needs to ensure they understand and have a plan in place to meet all requirements. The blame likely lies between those two roles.
      • by Rei ( 128717 )

        ESAB is a Swedish company. What use is it to NASA to dote largess on a Swedish welding firm?

        I'm actually rather disappointed with ESAB here. I have one of their MIG welders from the 1960s and it still works; they're a respectable name.

        I feel bad for NASA mind you, in that I don't think many of their problems are their own. They get all sorts of legacy systems forced upon them due to political reasons ("You can't do decision X that would be more efficient because 1000 people in my district would lose their

        • by Smask ( 665604 )

          ESAB is a Swedish company.

          ESAB was bought in 2012 by Colfax Corp and the manufacturing was moved to China.

    • by raymorris ( 2726007 ) on Friday April 17, 2015 @10:12PM (#49498373) Journal

      "and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.

          Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?

      We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.

      • by iluvcapra ( 782887 ) on Friday April 17, 2015 @11:25PM (#49498625)

        When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it.

        If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.

        Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.

        The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.

        • by jonwil ( 467024 )

          The SLS should be scrapped anyway, the only real reason it exists (at least in its current form) is to keep a bunch of contractors in key congressional districts in business after the end of the shuttle program.

        • by Anonymous Coward

          Musk hasn't destroyed *ANY* recovery barges, much less "two or three...in a row". In both failed landings, the barge suffered minimal (mostly cosmetic) damage.

          Most of the rest of what you posted is similarly accurate.

        • by itzly ( 3699663 )

          The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row

          The biggest reason is that he really wants to do a good job, because it's his own money and reputation.

          If a contractor overruns his budget, and the result is that he gets a bigger budget, where's the motivation to do a good job ?

          • "and the result is that he gets a bigger budget"

            Not always true. Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts. Sometimes, he gets sued but he government to recover the money. It totally depends on the cause.

            If a contractor overruns his budget because of unforeseen technical difficulties, that is one thing. They usually are on the cutting edge, so predicting that sort of thing is difficult. If they overrun their budget due to in
            • by itzly ( 3699663 )

              Not always true. Sometimes, he loses the contract and it goes to someone else

              That's a matter of skillful management. Make sure that the client is already invested too much, and that the overrun would cost less than finding someone else for the contract. Also, make sure that the original contract is written in a way that the budget is not guaranteed, and that you can blame the overrun on the client (poor specification). And finally, a nice bribe is always helpful.

            • by illtud ( 115152 )

              Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts

              2014 EU Procurement Directives now allow contracting authorities to take past failures into account when evaluating tenders.
              At last!

              https://www.gov.uk/government/... [www.gov.uk]

        • by Ecuador ( 740021 )

          If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed.

          I don't get your point. Isn't *THIS* a slip-up that is much worse than the SpaceX designed-to-be-expendable barges?

      • I dunno. If someone had been micromanaging it, maybe they would have remembered to reinforce the floor.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.

      I work for NASA. While I don't have direct involvement with the commercial crew program, major portions of it go on around me so I am familiar with how it works. (There was a 3 hour briefing on the various aspects of it in our department a few weeks back.)

      SpaceX does some cool stuff but the biggest reason they're able to do things so cheaply is that they have pretty lax testing standards. They lack the analysis experts NASA has; Lockheed does too, but have more experienced people. Everyone at SpaceX seem

      • by Anonymous Coward on Saturday April 18, 2015 @08:50AM (#49499971)

        Thanks for that.

        the biggest reason they're able to do things so cheaply is that they have pretty lax testing standards.

        These days, though, "lax testing standards" is in the eye of the beholder. I work for a Japanese manufacturer with a world-renowned reputation for quality.

        In the US, having the word "quality" in your job title is a career death sentence. In this Japanese company, it means that you are the créme de la créme.

        They look at testing as "black box monkey testing" at the end of the process. This is what they have done for multiple decades. They have Excel checklists that are thousands of lines long. If even one of those lines is not checked, or is anomalous, the whole shooting match comes to a screeching halt until said line is graced with a green check.

        I am trying to implement an inline "process quality" to their software development. You know, TDD, CI, CD and whatnot.

        My. God. It's damn near impossible.

        For one thing, they have a culture that deliberately sets up an antagonistic relationship between Quality and Engineering. Quality generally has more power. Engineering is considered to be a bunch of "yahoos," bent on degrading the Holy Quality.

        When Engineering managers (like me) try to suggest quality measures; even though these are not wild, "cutting edge" things, we are routinely dismissed and ignored, because we are obviously trying to avoid work and introduce bad quality.

        The simple fact of the matter is, that introducing the techniques I'm talking about could have a revolutionary effect on our quality. It would drastically reduce the cost of our production, and would certainly reduce (possibly to 0), the number of "time bomb" bugs that tend to slip through those massive checklists and explode in the face of influential people with high Twitter follower accounts.

        I strongly suspect that NASA has an extremely similar culture. The only way they can grok "quality," is through their 1950s-era "black box at the end" testing methodology.

        This has some real benefits (Read this to see what can happen when you have super-redundant black box testing and prototyping [ieee.org]).

        However, it is screamingly -UNBELIEVABLY- expensive and time-consuming (read: "expensive"). It also doesn't guarantee quality. Nothing really does.

        In the case of Apollo 13, NASA's anal prototyping and simulation exercises saved the astronauts. However, the preferred outcome would have been that the oxygen tank not have exploded in the first place. A lot more boring.

        I'm not sure that your statement was fair.

        Musk is an engineer. Quality folks see engineers as "out-of-control cowboys."

        I'm getting very, very sick of this highly insulting, and completely inaccurate portrayal.

      • by Anonymous Coward

        Then, there's a tendency towards analysis paralysis at NASA. Sometimes, it's better to just build it and try it, particularly with complex interacting systems where the analytical tools aren't available.

        NASA is the kind of place that if you were installing a FM radio in your lab to listen to music while you work, they would do a complete spectrum survey, consider 7 different kinds of antennas, develop theoretical models for the antennas and the building on which the antenna will be mounted, considering the

    • NASA has paid Musk Billions already.

    • by sopwath ( 95515 )

      This is exactly why the additional commentary at the end of the article is not needed.

      You imply that Elon Musk never made a mistake, in an effort to point out government "waste", while ignoring the fact that SpaceX made attempts to land a rocket twice and failed spectacularly.

      There's also the fact that NASA did roughly 60 years of research and development, mistakes and all, making it possible for Elon Musk to launch rockets so cheaply. SpaceX gets to skip all the expensive, embarrassing, "wasteful" parts of

  • ... is ten dollars on the shop floor and a hundred dollars in the field.

    Looks like they tried to save eleven dollars and got caught out!

  • by fuzzyfuzzyfungus ( 1223518 ) on Friday April 17, 2015 @10:03PM (#49498339) Journal
    The article does not mention where the cost of this error is going to fall. This seems like an important detail. On a sufficently complex project, one of the bevy of subcontractors fucking something up isn't a huge surprise; but I would be very, very, disappointed if NASA wasn't able to contract sufficiently vigorously to make the vendor eat the cost of delivering the goods as specified, rather than paying them for their effort no matter how well or badly they do.
    • It should be billed directly to the company, I'd imagine. That's what contracts are for.

      If you fail to fulfill a term of a contract, its your financial responsibility to fulfill that term.

      • And yet government contracts somehow continually manage to go over budget, and the government pays the cost instead of the contractor.
        • For most companies, the big government jobs would be a "bet the business" deal, and they would be bankrupt if they had to eat the cost of a mistake like this. No one will bid on the jobs except on a cost-plus basis.
          • by Imrik ( 148191 )

            To make it worse, larger companies will create smaller subsidiaries to bet on deals. That way if it fails the subsidiary won't be big enough to be able to complete the contract without an increase in the budget, forcing the government to pay them more.

  • Its nearly 3 inces. What was the precision of the Saturn 5?

    • by ebonum ( 830686 )

      Yep.

      degrees = 0.06
      radians = 0.06 * pi / 180 = 0.001047
      meters = 66.14 * TAN( 0.00104720 ) = 0.06926
      cm = 0.06926 * 100 = 6.926
      in = 6.926 / 2.54 = 2.727

      • Your result is correct, but your math is wrong. tan(0.06) = 0.06 * pi / 180 = 0.001047 66.14 * tan(0.06) = 66.14 * 0.001047 = 0.06926 = ~69mm However, 66.14 * tan(0.001047) = 0.00120861.
      • Ignore my other post, forgot break tags. Your result is correct, but your math is wrong.

        tan(0.06) = 0.06 * pi / 180 = 0.001047
        66.14 * tan(0.06) = 66.14 * 0.001047 = 0.06926 = ~69mm

        However, 66.14 * tan(0.001047) = 0.00120861.
        • P.S. I don't know if tan(0.06) is really equal to 0.06 * pi / 180, but they're the same to 8 decimal places, which is a weird coincidence.
      • I wonder how much that is in terms of weight distribution?

  • by dbIII ( 701233 ) on Friday April 17, 2015 @10:45PM (#49498481)
    Age old story of outsourcing - you still need to retain enough people to watch the contractors so they can't cut corners on the expensive bits.
    One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
    So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
    • I hate watching Grand Designs and then there's someone who has ordered three glass insulation windows from Sweden (most likely the case, it may just have happened like two times or whatever but still!) and then it either takes forever or there's wrong measurements / doesn't work.

      And then it has to be redone and then they are super-happy with the finished product because it turned out so good.

      BUT they didn't got their directly!
      (Sure windows may be one of those things which become more of a trouble in general

    • by sconeu ( 64226 )

      They should have known there was a problem when they found brown M&Ms in the break room candy dishes.

      • by illtud ( 115152 )

        +2 for:

        * an insightful comment
        * understanding the point of the oft-derided rider demand

    • Amazing! Only 1 person needed to speak up. It's hard for me to believe that not 1 single person noticed this and brought it up. Conspiracy money is that someone noticed it and it got swept under the rug. What does Occam and his razor say? I'd like to see the people on the project interviewed to confirm that there was not criminal conspiracy to cover this up.
      • by dbIII ( 701233 )
        One person who cared had to speak up and notice. Thus somebody with the goals of the project in mind instead of the goal of the contractor to have maximum profit.
        Outsourcing fails when you don't have enough people left to keep the contractors honest.
        • Yup. The relationship is rather adversarial in nature. NASA should have bribed the CEO of the outsourcing
          company to get it done right the first time.
          • by dbIII ( 701233 )
            As I've seen with the relationship between government and telecommunications companies giving the CEO an expensive bribe does not stop them screwing you over either.
            Paying someone to care just results in them caring about the money supply. Without "skin in the game" there is little reason to care.
      • Simple. There probably weren't many who knew both
        1) The floor needed reinforcing and
        2) It hadn't been done.

        Occam's Razor says communications error.

        Everyone actually in charge of building the thing would have believed the floor would be adequate. Whoever was in charge overall dropped the ball (maybe misread a statement to the effect "floor shall be reinforced to such-and-such-a-spec" as "floor has been reinforced to such-and-such-a-spec".

        • by dbIII ( 701233 )
          You can't just "forget" something so critical and conveniently avoid a very time consuming and expensive step.
    • by drwho ( 4190 )

      Many, many years ago I was a temp doing data entry for the sub-sub contractor for military night-vision goggles. the company was making the high-voltage power supplies. they had a QA spreadsheet in Lotus 123 that the results of QA test failures were supposed to be entered into, and because of bad 'programming', only the first 20 tests failures were tabulated, giving them results which showed a lower failure rate the more units they made. I pointed this out, was ignored, complained, was fired, tried to blow

      • I pointed this out, was ignored, complained, was fired, tried to blow the whistle, got no response.

        But you since gave up, because otherwise you'd have shared the name of the company here. Since you didn't, I think I'll just mark this one as "apocryphal" and move on with life.

        • by dbIII ( 701233 )
          So you are calling the above poster a liar? Nice.
          Do you do children's parties?
          • So you are calling the above poster a liar? Nice.

            Name and shame or you're a liar making shit up to make yourself seem more important. Put up or shut up is the name of the game, otherwise it's just FUD. Vague pronouncements might serve in government, but they don't suit here on Slashdot.

  • by drwho ( 4190 ) on Saturday April 18, 2015 @12:26AM (#49498789) Homepage Journal

    Actually, that's only sort-of true. I started with MCC interim release, but couldn't get it to work properly. So then I spent a few days downloading SLS and it worked just fine - well, as good as you could expect with only 4MB of ram. But I didn't notice any alignment issues, and I wasn't instructed to reinforce the floor so I didn't. I had problems with overheating during compilation though, which I fixed by a powerful floor fan pointed at the air intake of the PC. I later fixed this more gracefully with a home-made triple-sized heat sink. Maybe that's what NASA should do, build a giant heat sink onto it.

  • Just work with companies with a good record. I worked at a place that had a policy to hire minority and woman owned businesses. Some were OK, others couldn't perform to the needed standards.
  • Bullshit (Score:4, Insightful)

    by sexconker ( 1179573 ) on Saturday April 18, 2015 @01:08AM (#49498901)

    Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."

    Bullshit. The reason is they thought they could get away with it.

    • Of course it's completely impossible that somebody thought somebody else had done it.

      • It is. It takes an explicit effort to remove something from the contract when you're parceling out jobs to subcontractors. It takes explicit effort to remove a major component of the design in your diagrams, models, and plans. It takes an explicit effort to have no one on the staff say anything when it's blindly fucking obvious that thing your building is structurally unsound.

        • It takes an explicit effort to remove something from the contract when you're parceling out jobs to subcontractors.

          No, it just takes an editing mistake.

  • "The net effect on SLS’s development, both in terms of cost and possible delays"

    Sounds like the taxpayers are the ones who are going to get to pay for this "miscommunication" (see attempted fraud). Ah, "cost plus" contracts, you never cease to amaze. I hope someone waves this, that A-3 test stand debacle and all of the other "miscommunications" in the faces of all of the people trying to get money diverted from the CCDev program, a contract based program, to SLS.

    • by koan ( 80826 )

      Maybe that’s a way of generating profit when you have a government contract, make mistakes.

  • It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing

    No what's baffling is that the metric system is still not in wide use in the USA.

    That's baffling, a bunch of engineers screwing up is par for the course.

  • When a sub-contractor screws up, Apple has to pay.
  • Jet Line robotic welders never had this problem, think ISS. Is this another case of Congress throwing scrap meat to world market vendors and H1B hacks? Only to pay more for it later?
  • Even if NASA and ESAB had a "miscommunication" (I suspect an unresolved contract issue, which both sides thought the other has accepted responsibility for owning the floor contracting), what should have happened is that the ESAB equipment people, before starting work on the installation should have inspected the floor work they mandated to make sure it was done correctly. If this happened at all, you'd assume someone who notice that the floor has not been recently rebuilt AT ALL and would stop work until th

    • > An alternate, plausible chain of events is that NASA originally, disagreed with ESAB and felt the floor fix was unnecessary in the first place

      That's not plausible if you understand where the Michoud Assembly Facility is. It's next to New Orleans, and all of the ground around there is soft and swampy. If you are building anything heavy or that needs to be rigid, it needs heavy foundation reinforcement (thick slab, pilings, etc.).

  • This is why we can never have nice things under capitalism. I'm sure the contractor noticed this early in the construction process, and they took the gamble that maybe nobody would notice and saved some dollars. They got caught, and so it cost them... But if you think this is an isolated incident, you're sadly delusional. Pretty much every company out there pulls stunts like this, and most of them don't get caught. That's why they do it. Worship the all mighty ROI.

  • Given my knowledge of Boeing, the problem isn't with "quality"; the problem is with bad management -- and a culture of failure to admit that management can do no wrong. This can be easily exposed: Take a look at the jobs at Boeing. Look for a software tester / QA position. You will be lucky to find ONE. The jobs you do find are not test/qa; but rather development that can test their own code to the specifications written. And here is where QA comes in: what it there is a problem with the spec? And the

Everything that can be invented has been invented. -- Charles Duell, Director of U.S. Patent Office, 1899

Working...