Incorrectly Built SLS Welding Machine To Be Rebuilt 150
schwit1 writes A giant welding machine, built for NASA's multi-billion dollar Space Launch System (SLS), has to be taken apart and rebuilt because the contractor failed to reinforce the floor, as required, prior to construction: "Sweden's ESAB Welding & Cutting, which has its North American headquarters in Florence, South Carolina, built the the roughly 50-meter tall Vertical Assembly Center as a subcontractor to SLS contractor Boeing at NASA's Michoud Assembly Facility in New Orleans.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
ESAB was supposed to reinforce Michoud's floor before installing the welding tool, but did not, NASA SLS Program Manager Todd May told SpaceNews after an April 15 panel session during the 31st Space Symposium here. As a result, the enormous machine leaned ever so slightly, cocking the rails that guide massive rings used to lift parts of the 8.4-meter-diameter SLS stages The rings wound up 0.06 degrees out of alignment, which may not sound like much, "but when you're talking about something that's 217 feet [66.14 meters] tall, that adds up," May said.
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing, even though it was specified in the contract. It also suggests that the quality control in the SLS rocket program has some serious problems.
A penny saved (Score:1)
is a penny burned
Re: (Score:2)
It was totally the last CEO's fault. The $50million golden parachute was money well spent getting rid of that guy so we could hire someone competent.
Re: (Score:2)
Mer jobba, mer penga. Vad är om inte giller?
Re: A penny saved (Score:1)
Re: A penny saved (Score:1)
Re: (Score:2)
Cost plus contract. The more 'accidents' like this happen the more money the contractors get from the government teat.
Give the money to Elon Musk (Score:1)
The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.
Re: (Score:3)
Re:Give the money to Elon Musk (Score:4, Interesting)
Since I'm working in a large organization I have come to realize that the amount of documentation in many projects is huge - often so large that essential key information is masked away, or right out FUBARed.
It's also not uncommon that the customer requirements are "interpreted" by people with no technical knowledge whatsoever and they have a tendency to eradicate information that they think is "too technical", or information that they think drives unnecessary cost. Some people also have a tendency to rename things to a semantic that is to common people fuzzy. Even obfuscation occurs. At the same time documents are filled with a large number of pages listing old or discarded alternatives.
Re: (Score:2)
Re: (Score:2)
I suppose too many TL;DR situations where it really counted. Though, I suspect, it would indicate that the documents were probably more to cover ass than communicate in a clear and concise fashion the requirements. Then again, some people are good at writing and poor at communicating. Case in point: I was once given a few paragraphs describing logic requirements for a new functionality in an application, though since I need to convert it into code, I first converted it into a logic flow in English and sent
Re: (Score:3)
Re: (Score:3)
ESAB is a Swedish company. What use is it to NASA to dote largess on a Swedish welding firm?
I'm actually rather disappointed with ESAB here. I have one of their MIG welders from the 1960s and it still works; they're a respectable name.
I feel bad for NASA mind you, in that I don't think many of their problems are their own. They get all sorts of legacy systems forced upon them due to political reasons ("You can't do decision X that would be more efficient because 1000 people in my district would lose their
Re: (Score:1)
ESAB is a Swedish company.
ESAB was bought in 2012 by Colfax Corp and the manufacturing was moved to China.
have to rewrite muc federal law to not micromanage (Score:5, Informative)
"and not micromanage it". That's the rub. The micromanaging, the reporting and compliance costs, can be over 50% of the cost for some federal contracts, but most of the time that's required by thousands upon thousands of pages of federal law. When you have a comoany that knows how to do a certain thing , aka one of those evil corporations, getting hired by the federal government, some people want to do a lot of paperwork and stuff to keep track of what's going on, and other people go crazy with it. The organization I work for used to do a lot of federal contracts. We quit and now just do state contracts for states that are reasonable.
Still other people added a bunch of requirements for federal contracting that aren't really relevant to the project. For example, how many black women work for each of your major suppliers? How much do your interns make? Are all of the web pages and documentation you've ever made fully accessible to people who are both blind and deaf?
We quit dealing with the feds and certain states because it's just not worth it. It would cost SPACEX five times as much to build a federally-contracted rocket than it costs to build their own.
Re:have to rewrite muc federal law to not microman (Score:5, Interesting)
If we're doing something important, like killing Hitler, or trying to beat the Commies to the moon, federal procurement can be remarkably efficient. Clear goals, and the stated willingness to accept some waste as long as the job's done, can do that.
Unfortunately since about the mid-1970s (Watergate you say?!), approximately zero "waste," of any kind is tolerated on any federal project, as this is "profiteering" or "wasting the people's money," so a lot of contractor time is spent on compliance. This makes the process incredibly loss-averse, and probably too risk-averse to actually accomplish anything.
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row and he doesn't have to explain it to anybody but his accountant. If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed. Back in the late 50s NASA screwed up these kinds of operations all the time, but the American people tolerated it because of the Cold War. Nowadays the budget is so tight and public accountability is so fierce that frigging welding assembly subcontractors are apparently front-page news. We probably built and destroyed five facilities on the scale of this thing during Apollo and nobody batted an eye at the expense.
Re: (Score:1)
The SLS should be scrapped anyway, the only real reason it exists (at least in its current form) is to keep a bunch of contractors in key congressional districts in business after the end of the shuttle program.
Re: (Score:1)
Musk hasn't destroyed *ANY* recovery barges, much less "two or three...in a row". In both failed landings, the barge suffered minimal (mostly cosmetic) damage.
Most of the rest of what you posted is similarly accurate.
Re: (Score:2)
The reality is that Elon Musk is able to do a good job, because he can destroy two or three recovery barges in a row
The biggest reason is that he really wants to do a good job, because it's his own money and reputation.
If a contractor overruns his budget, and the result is that he gets a bigger budget, where's the motivation to do a good job ?
Re: (Score:2)
Not always true. Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts. Sometimes, he gets sued but he government to recover the money. It totally depends on the cause.
If a contractor overruns his budget because of unforeseen technical difficulties, that is one thing. They usually are on the cutting edge, so predicting that sort of thing is difficult. If they overrun their budget due to in
Re: (Score:2)
Not always true. Sometimes, he loses the contract and it goes to someone else
That's a matter of skillful management. Make sure that the client is already invested too much, and that the overrun would cost less than finding someone else for the contract. Also, make sure that the original contract is written in a way that the budget is not guaranteed, and that you can blame the overrun on the client (poor specification). And finally, a nice bribe is always helpful.
Re: (Score:2)
Sometimes, he loses the contract and it goes to someone else. Sometimes, he loses the ability to get awarded future contracts
2014 EU Procurement Directives now allow contracting authorities to take past failures into account when evaluating tenders.
At last!
https://www.gov.uk/government/... [www.gov.uk]
Re: (Score:2)
If the SLS had only one slip-up like that there would be a bloodbath of firings, senate hearings, press conferences with the President, and maybe the entire program might be scrubbed.
I don't get your point. Isn't *THIS* a slip-up that is much worse than the SpaceX designed-to-be-expendable barges?
Re: (Score:1)
Maybe we've inadvertently turned them into compliance engineers instead of rocket engineers. If they can make money by doing paperwork instead of science...
Re: (Score:2)
I dunno. If someone had been micromanaging it, maybe they would have remembered to reinforce the floor.
can be under emergency authority, but politically (Score:2)
That CAN be done under certain conditions, but unfortunately the political discourse of the time makes that politically expensive. For example, vice president Al Gore gave an award to one particular company and presented them as a case study of efficient and effective government contracting. He was right, they did a good job.
A few years later, when the Bush administration needed to have infrastructure rebuilt in Iraq, they turned to the same company. Since they were known to be good and they
Re: (Score:2)
Re: can be under emergency authority, but politica (Score:2)
Halliburton wasn't vilified because of the lack of paperwork. They were vilified as a part of the whole botched up business of rebuilding a country. They were also an integral part of the Pentagon ideal of subcontracting war waging and the support operations.
What they build really well though was the military-diplomatic complex a.k.a the embassy, which had the purpose to defend the occupiers from the angry natives. Tells a lot doesn't it?
Re: (Score:2)
Re: (Score:2)
Of course, when you write specification "X", there are usually some mistakes. A contractor may spot the mistakes, but follow the letter of the specification anyway. When everything is finished, and you discover that "X" was wrong, you get an additional contract to fix things.
Re: (Score:2)
And therein lies the problem - What needs to happen is give private company the goal X with cost X, and not give a shit how it gets accomplished.
That is not always the best approach, since they may still cut corners. The real solution, IMO, is for the original customer to do acceptance testing and ensure the contract has penalties for failures to meet requirements. Acceptance testing should be done in-house or a separate sub-contractor, though where skills exist the former would be better.
Re: (Score:2, Interesting)
The contractors that NASA has used have squandered and bilked NASA for all they are worth - If you give the goal of what the SLS does to SpaceX - (and not micromanage it) Elon Musk will have it completed in a much shorter amount of time.
I work for NASA. While I don't have direct involvement with the commercial crew program, major portions of it go on around me so I am familiar with how it works. (There was a 3 hour briefing on the various aspects of it in our department a few weeks back.)
SpaceX does some cool stuff but the biggest reason they're able to do things so cheaply is that they have pretty lax testing standards. They lack the analysis experts NASA has; Lockheed does too, but have more experienced people. Everyone at SpaceX seem
Re:Give the money to Elon Musk (Score:4, Insightful)
Thanks for that.
These days, though, "lax testing standards" is in the eye of the beholder. I work for a Japanese manufacturer with a world-renowned reputation for quality.
In the US, having the word "quality" in your job title is a career death sentence. In this Japanese company, it means that you are the créme de la créme.
They look at testing as "black box monkey testing" at the end of the process. This is what they have done for multiple decades. They have Excel checklists that are thousands of lines long. If even one of those lines is not checked, or is anomalous, the whole shooting match comes to a screeching halt until said line is graced with a green check.
I am trying to implement an inline "process quality" to their software development. You know, TDD, CI, CD and whatnot.
My. God. It's damn near impossible.
For one thing, they have a culture that deliberately sets up an antagonistic relationship between Quality and Engineering. Quality generally has more power. Engineering is considered to be a bunch of "yahoos," bent on degrading the Holy Quality.
When Engineering managers (like me) try to suggest quality measures; even though these are not wild, "cutting edge" things, we are routinely dismissed and ignored, because we are obviously trying to avoid work and introduce bad quality.
The simple fact of the matter is, that introducing the techniques I'm talking about could have a revolutionary effect on our quality. It would drastically reduce the cost of our production, and would certainly reduce (possibly to 0), the number of "time bomb" bugs that tend to slip through those massive checklists and explode in the face of influential people with high Twitter follower accounts.
I strongly suspect that NASA has an extremely similar culture. The only way they can grok "quality," is through their 1950s-era "black box at the end" testing methodology.
This has some real benefits (Read this to see what can happen when you have super-redundant black box testing and prototyping [ieee.org]).
However, it is screamingly -UNBELIEVABLY- expensive and time-consuming (read: "expensive"). It also doesn't guarantee quality. Nothing really does.
In the case of Apollo 13, NASA's anal prototyping and simulation exercises saved the astronauts. However, the preferred outcome would have been that the oxygen tank not have exploded in the first place. A lot more boring.
I'm not sure that your statement was fair.
Musk is an engineer. Quality folks see engineers as "out-of-control cowboys."
I'm getting very, very sick of this highly insulting, and completely inaccurate portrayal.
ANother NASA worker view (Score:1)
Then, there's a tendency towards analysis paralysis at NASA. Sometimes, it's better to just build it and try it, particularly with complex interacting systems where the analytical tools aren't available.
NASA is the kind of place that if you were installing a FM radio in your lab to listen to music while you work, they would do a complete spectrum survey, consider 7 different kinds of antennas, develop theoretical models for the antennas and the building on which the antenna will be mounted, considering the
Re: (Score:2)
NASA has paid Musk Billions already.
Re: (Score:1)
This is exactly why the additional commentary at the end of the article is not needed.
You imply that Elon Musk never made a mistake, in an effort to point out government "waste", while ignoring the fact that SpaceX made attempts to land a rocket twice and failed spectacularly.
There's also the fact that NASA did roughly 60 years of research and development, mistakes and all, making it possible for Elon Musk to launch rockets so cheaply. SpaceX gets to skip all the expensive, embarrassing, "wasteful" parts of
A dollar in design... (Score:2)
... is ten dollars on the shop floor and a hundred dollars in the field.
Looks like they tried to save eleven dollars and got caught out!
Re: (Score:2)
There is nothing wrong with outsourcing it. You just need to make sure you have good quality control. If it was boeing tasked with quality control they should be on the hook. I'm boggled why there wasn't a mufti-approach QC contract on this that used Engineers for quality assurance of the floor reinforcing so when they were left standing around doing nothing it would have been rather obvious something was missed.
This is basic resident engineer stuff. There is absolutely no reason whatsoever something in the
Re: (Score:2)
There is nothing wrong with outsourcing it. You just need to make sure you have good quality control.
Which drives costs up, often quite radically.
When you build something for your own company, the goal is to get as good quality as feasible within time and budget constraints. Next year's salary depends on it.
When you build something as a bidding contractor for the government, the goal is to reduce your costs by as much as you can get away with and exceeding the budget with as much as you can get away with.
It doesn't matter if what you deliver is utter crud as long as you can get away with it. Politicians
Re: (Score:3)
Re: (Score:2)
Indeed, the figures Musk cited a couple years ago was that over 80% of the part count of a Falcon 9 is sourced in-house; it's a critical part of their approach to keeping costs down. He wanted to do that with Tesla as well but it proved impossible, only about 20% of their parts (at the time) were produced in-house. Unsurprisingly the biggest problems in their early days came from external suppliers, like the gearbox issue on the Roadster.
Re: (Score:3)
Rather the opposite. WIth the "libtards" truly in charge, there would be no outsourcing and subcontracting, and NASA would hire people to build things themselves.
The republicans are the ones that demand outsourcing and paperwork that often equals half the total costs. Because heavens forbid if a government agency did something that private companies could do. That is considered anticompetitive theft by the right. Which is why NASA can't do much themselves anymore, and get less bang for the buck.
Re: (Score:1)
Which is why NASA can't do much themselves anymore, and get less bang for the buck.
You do realize that NASA never 'did much themselves', and almost all their hardware manufacturing has always been contracted out, right?
Or were you just trolling?
The real problem I see with SLS is that it's a rocket without a mission that no-one expects to fly more than a couple of times, so there's no real incentive to do a good job.
The important question... (Score:4, Interesting)
Re: (Score:2)
It should be billed directly to the company, I'd imagine. That's what contracts are for.
If you fail to fulfill a term of a contract, its your financial responsibility to fulfill that term.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
To make it worse, larger companies will create smaller subsidiaries to bet on deals. That way if it fails the subsidiary won't be big enough to be able to complete the contract without an increase in the budget, forcing the government to pay them more.
It does add up (Score:2)
Its nearly 3 inces. What was the precision of the Saturn 5?
Re: (Score:2)
Yep.
degrees = 0.06
radians = 0.06 * pi / 180 = 0.001047
meters = 66.14 * TAN( 0.00104720 ) = 0.06926
cm = 0.06926 * 100 = 6.926
in = 6.926 / 2.54 = 2.727
Re: (Score:1)
Re: (Score:1)
tan(0.06) = 0.06 * pi / 180 = 0.001047
66.14 * tan(0.06) = 66.14 * 0.001047 = 0.06926 = ~69mm
However, 66.14 * tan(0.001047) = 0.00120861.
Re: (Score:1)
Re: It does add up (Score:1)
Not a coincidence. sin(x) ~= tan(x) ~= x for small x.
Re: It does add up (Score:2)
I wonder how much that is in terms of weight distribution?
Re: (Score:2)
The final inspection yes, the in process QC not so much.
Age old story of outsourcing (Score:5, Interesting)
One blatant example I saw was with non destructive testing of welds in high pressure pipework leading to portions of a turbine in a coal fired power station. At those welds it was done by spraying on thing white paint, using a magnet and spraying on a fluid with suspended magnetic "dust" that would collect wherever defects disrupted the magnetic field. Access was a bit tight so the contractors tested the top of the pipes and they ran the magnet around the bottom of the pipe without looking at it so that some scratches would be left to show that the magnet had been used. The lazy pricks were caught doing that so we had to send someone along as an observer and make them do a couple of weeks worth of work over again, because with their scratch trick we had no way of knowing is any inspection had actually been done or not.
So MBA types - that person standing off to the side not doing anything during a concrete pour may be there solely to reduce fuckups due to dishonest contractors.
It's shameful (Score:1)
I hate watching Grand Designs and then there's someone who has ordered three glass insulation windows from Sweden (most likely the case, it may just have happened like two times or whatever but still!) and then it either takes forever or there's wrong measurements / doesn't work.
And then it has to be redone and then they are super-happy with the finished product because it turned out so good.
BUT they didn't got their directly!
(Sure windows may be one of those things which become more of a trouble in general
Re: (Score:3)
They should have known there was a problem when they found brown M&Ms in the break room candy dishes.
Re: (Score:2)
+2 for:
* an insightful comment
* understanding the point of the oft-derided rider demand
Re: (Score:1)
Re: (Score:2)
Outsourcing fails when you don't have enough people left to keep the contractors honest.
Re: (Score:1)
company to get it done right the first time.
Re: (Score:2)
Paying someone to care just results in them caring about the money supply. Without "skin in the game" there is little reason to care.
Re: (Score:2)
Simple. There probably weren't many who knew both
1) The floor needed reinforcing and
2) It hadn't been done.
Occam's Razor says communications error.
Everyone actually in charge of building the thing would have believed the floor would be adequate. Whoever was in charge overall dropped the ball (maybe misread a statement to the effect "floor shall be reinforced to such-and-such-a-spec" as "floor has been reinforced to such-and-such-a-spec".
Re: (Score:2)
Re: (Score:2)
Many, many years ago I was a temp doing data entry for the sub-sub contractor for military night-vision goggles. the company was making the high-voltage power supplies. they had a QA spreadsheet in Lotus 123 that the results of QA test failures were supposed to be entered into, and because of bad 'programming', only the first 20 tests failures were tabulated, giving them results which showed a lower failure rate the more units they made. I pointed this out, was ignored, complained, was fired, tried to blow
Re: (Score:2)
I pointed this out, was ignored, complained, was fired, tried to blow the whistle, got no response.
But you since gave up, because otherwise you'd have shared the name of the company here. Since you didn't, I think I'll just mark this one as "apocryphal" and move on with life.
Re: (Score:2)
Do you do children's parties?
Re: (Score:2)
So you are calling the above poster a liar? Nice.
Name and shame or you're a liar making shit up to make yourself seem more important. Put up or shut up is the name of the game, otherwise it's just FUD. Vague pronouncements might serve in government, but they don't suit here on Slashdot.
I started with SLS, but then switched to Slackware (Score:4, Funny)
Actually, that's only sort-of true. I started with MCC interim release, but couldn't get it to work properly. So then I spent a few days downloading SLS and it worked just fine - well, as good as you could expect with only 4MB of ram. But I didn't notice any alignment issues, and I wasn't instructed to reinforce the floor so I didn't. I had problems with overheating during compilation though, which I fixed by a powerful floor fan pointed at the air intake of the PC. I later fixed this more gracefully with a home-made triple-sized heat sink. Maybe that's what NASA should do, build a giant heat sink onto it.
Low Bid Equals Low Quality (Score:1)
Bullshit (Score:4, Insightful)
Asked why ESAB did not reinforce the foundation as it was supposed to, May said only it was a result of "a miscommunication between two [Boeing] subcontractors and ESAB."
Bullshit. The reason is they thought they could get away with it.
Re: (Score:2)
Of course it's completely impossible that somebody thought somebody else had done it.
Re: (Score:2)
It is. It takes an explicit effort to remove something from the contract when you're parceling out jobs to subcontractors. It takes explicit effort to remove a major component of the design in your diagrams, models, and plans. It takes an explicit effort to have no one on the staff say anything when it's blindly fucking obvious that thing your building is structurally unsound.
Re: (Score:2)
No, it just takes an editing mistake.
Re: (Score:3)
Had there been an "architect" or "main designer"
That would be Boeing's role (as the general contractor). Boeing is really good about managing things that they understand, like building airplanes. Because there's usually some geezer on the payroll that remembers how they did things 'back in the old days'. But not so good at doing the one-off type projects. They basically broker services and talent between the various subcontractors that actually know how to do the work.
And who's going to pay for this "miscommunication" (Score:2)
"The net effect on SLS’s development, both in terms of cost and possible delays"
Sounds like the taxpayers are the ones who are going to get to pay for this "miscommunication" (see attempted fraud). Ah, "cost plus" contracts, you never cease to amaze. I hope someone waves this, that A-3 test stand debacle and all of the other "miscommunications" in the faces of all of the people trying to get money diverted from the CCDev program, a contract based program, to SLS.
Re: (Score:1)
Maybe that’s a way of generating profit when you have a government contract, make mistakes.
Laugh... (Score:1)
It is baffling how everyone at NASA, Boeing, and ESAB could have forgotten to do the reinforcing
No what's baffling is that the metric system is still not in wide use in the USA.
That's baffling, a bunch of engineers screwing up is par for the course.
And I thought (Score:2)
If You Want It Done Right, Buy Swed? Ouch. (Score:2)
Re: (Score:2)
How this should have been prevented... (Score:2)
Even if NASA and ESAB had a "miscommunication" (I suspect an unresolved contract issue, which both sides thought the other has accepted responsibility for owning the floor contracting), what should have happened is that the ESAB equipment people, before starting work on the installation should have inspected the floor work they mandated to make sure it was done correctly. If this happened at all, you'd assume someone who notice that the floor has not been recently rebuilt AT ALL and would stop work until th
Re: (Score:2)
> An alternate, plausible chain of events is that NASA originally, disagreed with ESAB and felt the floor fix was unnecessary in the first place
That's not plausible if you understand where the Michoud Assembly Facility is. It's next to New Orleans, and all of the ground around there is soft and swampy. If you are building anything heavy or that needs to be rigid, it needs heavy foundation reinforcement (thick slab, pilings, etc.).
Come and see the violence inherent in the system! (Score:2)
This is why we can never have nice things under capitalism. I'm sure the contractor noticed this early in the construction process, and they took the gamble that maybe nobody would notice and saved some dollars. They got caught, and so it cost them... But if you think this is an isolated incident, you're sadly delusional. Pretty much every company out there pulls stunts like this, and most of them don't get caught. That's why they do it. Worship the all mighty ROI.
"Quality control" (Score:2)
Re: (Score:2)
Maybe the layed off people from MakerBot went to work for ESAB
http://www.engadget.com/2015/04/17/makerbot-lays-off-20-percent-of-its-employees/
Re: (Score:2)