Elon Musk: Faulty Strut May Have Led To Falcon 9 Launch Failure 220
garyisabusyguy writes: This Forbes article provides the best analysis of the loss of the last Falcon 9 mission based on information released by Elon Musk to reporters. Highlights include:
- 1. Sound triangulation led them to identify a strut holding helium tank as root cause where the falling helium tank pinched a line causing overpressure in the LOX tank.
- 2. The failure occurred at 2,000 pounds of force, and the struts were rated at 10,000 pounds of force. They initially dismissed this as a cause until sounds triangulation pointed back to the strut
- 3. Further testing of struts in stock found one that failed at 2,000 pounds of force, with further analysis identifying poor grain structure in the metal, which caused weakness
- 4. It will be months before the next launch while SpaceX goes over procurement and QA processes all struts and bolts, and re-assesses any "near misses" with Air Force and NASA
- 5. Next launch will include failure mode software, which will allow recovery of the Dragon module during loss of the launch vehicle since they determined that it could have saved the Dragon module in this lost mission
.
Holy Jebus (Score:5, Funny)
Now THAT is how you summarize.
Re:Holy Jebus (Score:5, Interesting)
Now THAT is how you summarize.
Well, missing from TFS was what SpaceX is doing to prevent this specific problem in the future:
To avoid this type of accident occurring in the future, the company will now individually test every strut it installs on a Falcon 9, regardless of its material specification. It's also considering a different material for its bolts, as the bolthole was the likely site of failure, and will likely switch strut suppliers.
The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification. Accept the universal truth that the vendor is a lying bastard, test as needed, and get on with life. If SpaceX ever reaches their reusability goal, the cost of all the testing will be spread across many flights anyway.
Re: (Score:3)
Yeah, I tried to cover that under item 4 with "goes over procurement and QA processes all struts and bolts"
Admittedly not a "smoking gun" sort of statement, just *ahem* a summary
Re: (Score:2)
Couldn't last forever, like any proper winning streak, and often times more is learned from one's failures.
Carry on Elon, and may the odds be ever in your favor.
Re: (Score:2)
Ah yes, the Gregory House/Fox Mulder method: everybody lies, trust no one.
Re: (Score:3)
Ah yes, the Gregory House/Fox Mulder method: everybody lies, trust no one.
That's certainly a lot safer than the alternative.
Re: (Score:2)
Fairy tales begin "once upon a time".
War stories begin "this shit really happened"
Specs begin "1.0".
Re: (Score:2)
Actually, war stories begin "this is no shit. There I was..."
Re: (Score:3)
That reminds me of the joke about the boy scout who would test all his matches and save the ones that lit successfully.
Re:Holy Jebus (Score:5, Insightful)
Elon is surely really fuming about this one, as I know from past interviews with him that he really doesn't like having to source hardware from outside suppliers. He has the old "robber baron" mindset of wanting to get the whole production chain start-to-finish in house, and it's one of the things that really frustrated him when he started Tesla: at the time of the last interview I read on the subject (something like 3 or 4 years ago), he had gotten SpaceX up to 80% in-house, but Tesla was only up to 20% in-house. Car manufacture has long been all about sourcing parts from a wide range of outside suppliers.
But even at 80% in-house at SpaceX, looks like that remaining 20% still bit them : Seriously, failing at 1/5th the rated failure value? The vendor might as well have given them a cardboard cutout with the word "strut" written on it in sharpie.
Re:Holy Jebus (Score:5, Interesting)
Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.
Sad that the Falcon Heavy won't be launched until next spring, I've been really looking forward to that. Oh well...
Re: (Score:3)
Also, maybe it's just because I've never worked in that industry before, maybe it's common practice in rocketry, but is anyone else impressed with the use of sound triangulation to figure out which part broke? I've never heard of that being done before.
*camperdave raises his hand.
I am impressed that they were able to do this. I imagine that this would be difficult under the best of circumstances, but inside a rocket during a launch? That's not the type of environment I would expect to hear anything apart from the roar of the engines. I'm picturing 3D renderings showing simulated sound waves, experts trying to line up similar sonic wave-fronts to compute timings, and lots of computer time.
Re: (Score:3)
Real life is the highest fidelity 3D simulation!
I'm stealing this for my sig.
Re:Holy Jebus (Score:5, Informative)
Using sound to probe structures, materials and devices is a pretty ancient technique, it's just advanced with the tools available. Stonemasons used to check blocks of stone to locate faults by tapping on them and listening for sounds that would indicate cracks or other faults. One of my grandfathers tried to teach me how to listen for potential rot zones in wooden walls when I was a kid also. Now, with modern tools, we can just do it much faster, and with more precision.
Re: (Score:2)
It is one thing to use such techniques to test something in a quiet area like a testing chamber, and something completely different to use this technique not only in a rocket that is accelerating at 32 m/s^2 but also has a whole bunch of other noise going on from nine operating turbo pumps, the rocket engines themselves, and other things rattling around tied to that whole system. In addition, to be able to locate a cause while the whole rocket is undergoing massive unplanned disassembly (aka an explosion r
Re:Holy Jebus (Score:4, Interesting)
Such techniques are used in live environments today, such as factories, oil platforms etc. In fact, I'd be surprised if SpaceX DIDN'T have at least 6+ microphones or other vibration sensors relaying telemetry, and baselines from previous launches to compare with. With that baseline, you can tell if it happens to be a turbopump that malfunctions. Hell, if a strut or something would break in my brothers boat, we hear it immediately, because the overall vibration and thus overall boat noise will be altered.
Also, they clearly have decent bandwidth to the rocket during launch, given that they can have video feeds etc, and you can easily get multiple audio telemetry feeds to take less bandwidth than even a low-res video feed.
Re: (Score:3)
Re: (Score:2)
Re:Holy Jebus (Score:4, Interesting)
Re: (Score:2)
Gah, thanks for reminding me... My father being a train buff, and his father working on the railroad for 45 years, I should have remembered to bring that up.
It's not just audio triangulation (Score:2)
The sound triangulated was in cryogenic liquid oxygen at 50 PSI. The speed of sound in that is approximately 1 kilometer per second.This paper [nasa.gov] is about calculating the exact speed. Elon talked in the conference about reading telemetry with millisecond accuracy. But this would yield only 1 meter resolution.
Re: (Score:2)
Re: (Score:3)
I have a friend who helped design the Tesla drive train. He said they do so much in house since they have a lot of problems with components from 3rd parties, especially China. Look at all the transmission failures the Roadster had until they did their own in-house design.
It also allows for them to make changes quickly. Apparently they make tweaks to the design of the car almost every week which would be impossible to do with a heavy reliance on 3rd parties for manufacturing.
Re: (Score:2)
But even at 80% in-house at SpaceX, looks like that remaining 20% still bit them : Seriously, failing at 1/5th the rated failure value? The vendor might as well have given them a cardboard cutout with the word "strut" written on it in sharpie.
Indeed. I wonder if SpaceX can sue the supplier? I wonder what the contract terms were.
Re: Holy Jebus (Score:2)
It sounds like it was a rebranded Kerlington Model Rockets and Paper Products Inc. strut.
"The EAS-4 Strut Connector (frequently referred as "space tape") is a part that allows linking two parts together..."
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
That is, sort of, an argument for TCP. But UDP is often preferred nonetheless... And increasingly so even for the traditionally TCP-applications, such as file-transfers [sourceforge.net].
Re: (Score:2)
"The sourceforge.net website is temporarily in static offline mode. Only a very limited set of project pages are available until the main website returns to service."
Sound familiar, anybody?
Re: (Score:2)
Sourceforge is blocked by chrome and ublock and everybody sane. Stop using it.
Re: (Score:3)
The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification.
Spacex, welcome to NASA's world.
I'm rooting for them for sure, but After learning the hard lessons, and the inevitable expenses to correct them, their costs will increase. dramatically.
Re: (Score:2)
I'm rooting for them for sure, but After learning the hard lessons, and the inevitable expenses to correct them, their costs will increase. dramatically.
Yeah, because verifying that subcontractors built components they way they should will totally add half a billion dollars to the cost of each launch.
Besides which, once they start reusing stages, they'll know they're good because they've already flown once.
Re: (Score:2)
I'm rooting for them for sure, but After learning the hard lessons, and the inevitable expenses to correct them, their costs will increase. dramatically.
Yeah, because verifying that subcontractors built components they way they should will totally add half a billion dollars to the cost of each launch.
Besides which, once they start reusing stages, they'll know they're good because they've already flown once.
Nw expand that to all the parts
. Which part is going to fail? It's not a trap per se, but rockets have not changed one fundamental aspect, and that is they are sticks with a controlled explosion/deflagration coming out one end, and they stress the bejabbers (technical term) out of every single part in them.
Light enough to fly, strong enough to withstand crushing forces. Barely compatible requirements.
Re: (Score:2)
Nw expand that to all the parts
They make most of the parts themselves, so presumably they're already doing at least random testing of those parts. Even if they did test them all, it won't add half a billion dollars to the cost of each launch, nor will they be testing many of them before reusing a stage which worked fine last time.
You have to try really, really, really hard to make your launcher anywhere near as expensive as SLS.
Re:Holy Jebus (Score:5, Informative)
Nw expand that to all the parts
You mean expand it to the fraction of the 20% outsourced parts that weren't already being exhaustively tested? Most of those outsourced parts are electronic, because Elon Musk doesn't own a semiconductor factory (yet). All of those electronic parts are tested exhaustively and repeatedly, right up until the literal second of launch, as software verifies sanity in all systems. What's left are things like struts, that should be stupid-simple enough to be trustworthy. And apparently aren't.
So some fraction of a fraction of the parts will need newly exhaustive testing. Not free, but really, it's not that big of a bill.
Re: (Score:2)
m rooting for them for sure, but After learning the hard lessons, and the inevitable expenses to correct them, their costs will increase. dramatically.
I don't think that reuse rockets will be considered prime rockets. I expect they'll be used for lesser-importance launches if there's any real risk of a fault developing that couldn't be detected.
Re:Holy Jebus (Score:5, Interesting)
"the vendor is a lying bastard"
As a former aerospace systems engineer (now RF systems engineer), I found that that is certainly how the US Government and their Prime Contractors treat their suppliers. Every process in place is there to make sure you're not lying about something, cutting a corner, or inflating expenses. And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.
So even if you are one of those vendors that acts in good faith, and believes that a quality product is the best advertising, and that if your project is paid for by tax money then you're ethically obligated to do the best job possible, you get treated exactly like the asshole who made the decision to make a few extra bucks profit by not properly verifying the workmanship on a major structural component for a vehicle that he knew would eventually be manned.
And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.
Re: (Score:2)
And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.
Eventually, every young engineer realizes "wait, we're a vendor too", and his eyes are opened. It's a formative moment towards the end of his apprenticeship, no?
Re: (Score:2)
In a koan:
Eventually, every young engineer realizes "wait, we're a vendor too", and... the young engineer is enlightened.
Re:Holy Jebus (Score:5, Informative)
And the depressing thing is that every one of them is there to prevent recurrence of a dishonesty that actually took place in the past.
And sometimes it's not even dishonesty, just stupidity. This was probably more true in the earlier days of aerospace. In a book about the design and construction of the Lunar Module (I think it's Chariots for Apollo, but could be wrong) there's a section on how many of the subcontractors had to be taught clean-room and quality techniques. There's one episode where one of the Grumman managers goes out to some paint pigment company who happened to get the contract for the silver-zinc LM batteries (because they had supplies of the right materials) and sees the batteries being assembled -- in a dirty shed by people who are smoking cigarettes while doing the assembly. (They threw out the entire batch, trained everyone how to do things the aerospace way, and set up a clean room, and AFAIK there was never a problem with the LM batteries.)
On the other hand the ladies sewing space suit pressure garments at the Playtex Girdle factory knew the astronauts' lives depended on what they were doing, and did it right the first time.
Re: (Score:2)
And the most depressing thing about this?
It basically excludes new players from the market. Only the big firms have the resources to go through all that compliance paperwork. Which means the only people left are the lying, cheating, scum that caused the problem in the first place.
This has only fostered a risk-averse mentality that chokes every aspect of government and big business. No wonder it's the small firms that have the reputation for innovation - it's because they still have their innocence and aren'
Engineering and errors (Score:3)
And as an engineer, I know that most engineers want to act in good faith. Some are inept or inexperienced but they still have good faith. The problem lies in management. Once you get the lawyers and bean counters involved is when asshole decisions like that get made.
I'm an engineer but I'm also an accountant (aka a bean counter). I also am management in my company and I'm in charge of the engineering, production and accounting among other things. You are absolutely right that most of the time the fault for most failures ultimately is due to management decisions. At the end of the day the buck stops with them and that is how it should be. HOWEVER, management ultimately relies on the expertise of engineers and the reasoned opinions of those who report to them. If ma
Re: (Score:2)
Geez folks, it's a flying, semi-autonomous/autonomous vehicle..... even if it's 1960's tech:
test,test,test,test.
Simulation and spec reports can only get you so far.
Re: (Score:2)
The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard".
Hardly. This points more to poor vendor selection and a crap procurement process. In many cases the costs of faults should be contractually passed back to the vendor. The end result is a more costly product but with higher quality as a result.
The problem is that salesmen are lying bastards. When you get the technical people on the phone (if you can get them) the complete and typically trustworthy story comes out.
Re: (Score:2)
The first thing any engineer (in any discipline) needs to learn when starting a real job is "the vendor is a lying bastard". I think it will work out substantially cheaper in the long run to test every strut rather than to go crazy with the material specification. Accept the universal truth that the vendor is a lying bastard, test as needed, and get on with life. If SpaceX ever reaches their reusability goal, the cost of all the testing will be spread across many flights anyway.
The second thing any engineer needs to learn is cost-benefit analysis:
1. I always choose the lowest bidding vendor, and he is always a lying bastard who can't deliver on spec, on time, or on budget.
2. Testing every part or losing rockets costs a lot of money than I saved on the lowest bidder.
3. Maybe I should vet my bids more carefully with plant visits, spot checks, and intermittent testing. Then choose the best vendor and not be cheapest one.
This approach will end up saving you money with high-visibility,
Re: (Score:3, Funny)
Is that how you itemize? (Score:2)
By manually numbering each item of <ul> instead of simply using <ol>?
Re: (Score:2)
Here's what a numbered list looks like in slashcode:
You really should stop leaping to conclusions - it can cause you to fundamentally misunderstand the world you live in, which would explain a lot of the stuff you post here ;)
That was easy (Score:5, Funny)
So after weeks of investigation it turns out it's a failure mode that even the most amateur of KSP players recognize.
When in doubt, add more struts (Score:5, Funny)
They should have added a lot more of them, clearly. It's not like struts have any mass.
Re: (Score:2)
But with that many extra pieces they will have to slow down time during launch to make sure the physics don't get wonky.
Re: (Score:2)
Along with the obvious solution - add more struts, lots more, can't have too many struts...
(to be fair, most if not all, ksp players have crashed and burned way more often than SpaceX)
Re: (Score:2)
I crash and burn more often per half hour of play than SpaceX has so far.
Something is fishy (Score:3)
the heilum bottle would have shot to the top of the tank at high speed
That sounds a lot different than "a hose may have been pinched" Has anyone been able to find audio of the actual conversation?
Re: (Score:2)
Re: (Score:3)
This one seems to be a pretty decent explanation: http://www.orlandosentinel.com... [orlandosentinel.com]
Heh. One bit of that article made me chuckle (emphasis mine):
The explosion destroyed the $70 million rocket, its Dragon I capsule and 4,000 pounds of supplies that was headed to the International Space Station. None of it was insured.
I can just imagine Elon going to his local insurance agent, trying to get coverage for an experimental vehicle carrying 400 tons of rocket fuel and LOX.
Re: (Score:3)
Lloyd's of London will insure anything. You might just not like the premiums.
In this case, self insurance makes sense.
Re: (Score:2)
Man, every time I hear people talking about LOX like that, I think they must have one hell of a lot of bagels on the ISS.
Interesting way to sabotage SpaceX (Score:5, Interesting)
Or faulty parts made of metal.
It's just a thought, but would a competitor stoop to that? Even if not now, at some point in the future?
Re:Interesting way to sabotage SpaceX (Score:5, Informative)
Similar business practices have been used in the past. Rockefeller is a well-known example: He was an obsessive monopolist, unable to stomach the existence of any competitor to his Standard Oil empire. Among the tricks he used was to buy up manufacturers of components used in oil drilling and refining, and then refuse to sell replacements to competitors - driving them out of business when their expensive industrial machines eventually broke down and couldn't be repaired. With the competitor driven out of business Rockefeller easily purchased what was left and incorporated it into his company. He sabotaged one company by thus denying them access to oil-carrying carts for trains - and when they switched to shipping in barrels, he purchased the one company that could make a barrel sealing compound compatible with crude oil and altered the formulation to make it chemically unsuited.
Rockefeller's business practices went down in legend - you can thank him for modern antitrust laws: The first ones in the US were passed expressly in order to target him. He was the Bill Gates of the 1800s: Built up a fortune through unethical and at times outright illegal business practices, only to eventually retire and spend the rest of his life giving it away in huge grants to charitable causes.
It would be a lot harder to pull something like that today though - there are stricter regulations and laws against such things. It could be done, but it would need a great deal of legal caution and skill to avoid liability, or at least to ensure the liability lay with a collapsing shell company.
Re: (Score:3, Funny)
Similar business practices have been used in the past. Rockefeller is a well-known example: He was an obsessive monopolist, unable to stomach the existence of any competitor to his Standard Oil empire. Among the tricks he used was to buy up manufacturers of components used in oil drilling and refining, and then refuse to sell replacements to competitors - driving them out of business when their expensive industrial machines eventually broke down and couldn't be repaired.
Sounds like the invisible hand of the free market in action.
Re: (Score:2)
It's just a thought, but would a competitor stoop to that? Even if not now, at some point in the future?
John Broder volunteered.
Re:Interesting way to sabotage SpaceX (Score:5, Interesting)
I believe they would. I worked for American Rocket Company in 1989. They developed hybrid engine IP that SpaceDev now owns.
Anyhow, in the late 80's the owners of what would become AmRoc decided to go into the launch business and build a single stage suborbital proof of concept vehicle. Oxidizer was helium pressurized LOX tank and steered by the injected fuel vector control (60% Hydrogen Peroxide, just a few gallons). It needed a guidance system. I was a new grad hired to help obtain it, and do some of the wiring and testing.
The vendor was in Boulder CO. I forget their name. The name no longer exists because AmRoc's chief competitor, Orbital Sciences Corporation sabotaged AmRoc's avionics contract. How? On the eve of our CDR with this Boulder company, all of the employees quit, except for the two owners and a single tech who'd started the company with them. Why did they quit? Orbital Sciences Corp (now Orbital ATK, after they acquired Alliance earlier this year) hired every one of them out from under their feet promising to open a Boulder office.
And it worked. Instead of a custom designed flight computer, gyros, telemetry, data acquisition and etc., we got an avionics system made from secondhand Japanese gyros, engineering model electronics left at the Boulder shop, and the rest from the Omega catalog. Furthermore, only two gyros were available, and in that case, the Z-axis is the only choice to go uninstrumented. Which meant that the flight profile would have to rely on a simple timer schedule: when to start steering away from vertical, when to separate payload, etc. And when to pull the umbilical cord.
All of these things contributed to the failure mode: which was that the LOX valve (a 2.5" gate valve) became encased in ice after frost from the previous day's rehearsal/test pooled around the valve and then refroze when the LOX was filled on launch day. It actually opened about 10%... enough to light the engine. And the person in charge of manually pulling the umbilicals (a payload customer, not an employee) jumped the gun and didn't wait for visual verification of liftoff... so no command could be given to close the valve and turn off the engine. As a result, the rocket sat on the pad and idled, the timer ran up to the moment when the thrust vector Peroxide started flowing... and the X and Y accelerometers saw no response... so more peroxide flowed. Until it pooled in the flame bucket and caught fire.
We sent a nice big black cloud over Santa Maria that day. You can read about it here. [latimes.com] But we did prove how safe a hybrid is: if it had been a solid engine or liquid fueled rocket, there would have been a very large explosion. Instead we effectively had a tire fire.
Amroc laid off 90% of its employees within 2 months. Closed its doors and sold its IP to Westinghouse a couple of years after that. Westinghouse later sold it to SpaceDev. And some very happy very well paid engineers in Boulder Colorado earned some very bad karma.
Do I think that Orbital ATK would pay a third party vendor to skimp a little on Acceptance Testing of critical structural components made for SpaceX?
Why yes. Yes I do.
Transparency (Score:4, Insightful)
I am amused by the fact that a private company does better than our government at disaster transparency. That said, it is pretty stupid that Space X has not been testing random parts to confirm they meet the requested specifications. Spec verification is a basic part of outsourcing. All outsourcing fails if you can't verify that you're getting what they promised you.
Re: (Score:2)
You can test random struts, but you can't test ALL struts or you're left with no struts. Sounds like they didn't test the right ones is all.
Re: (Score:2)
You can test random struts, but you can't test ALL struts or you're left with no struts. Sounds like they didn't test the right ones is all.
Who says you have to test them to failure? Just test them for the designed load.
Re: (Score:2)
I'm very much not an expert with this stuff, but it seems to me that if a particular strut is going to fail at 10,001lbs and you test it to 10,000lbs, it might pass the test but you still don't want to reuse it. There has to be some cutoff point where you're causing long term damage, but it's still functioning to requirements, right?
Re: (Score:3, Interesting)
Conveniently, this part has about a 5x margin (it was implied elsewhere that the 2000lbf is also right about the design load). They could test to two or three times the design load and not even be close to the rating.
Re:Transparency (Score:5, Insightful)
You can test random struts, but you can't test ALL struts or you're left with no struts. Sounds like they didn't test the right ones is all.
Quite true, but there are OTHER ways of figuring out the strength of the part other than testing them to failure or causing damage to the parts. Some of these methods are quite expensive, but effective. How you get a one part that's 1/4th the strength the design requires through a manufacturing process, onto a flight ready hardware assembly and not know it says serious problems lurk within. Seems Space-X has some defects in it's quality assurance processes, and that should scare you more than this one launch failure. They are playing way too fast and loose with quality and I'd be very worried about their efforts towards getting "human" rated are not going well.
This was a structural failure. A failure that can likely be traced to a part that was too weak for the designed loads. This isn't a DESIGN failure, it's a QUALITY failure, and that puts the whole program into question. What has to happen now is that the whole QA process needs to be revisited and revamped to prevent structural components from sneaking though which are not strong enough to do what they are designed to. THEN you have to go though your whole stock of parts, sub assemblies, and flight ready assemblies and figure out what you can verify as trustworthy using your NEW QA process, throw out the rest and order, assemble and test replacements.
This isn't an easy or quick fix...
Then there is the whole, our supplier lied to us, approach, which will be quicker to deal with, but only because you just have to obtain a batch of replacement struts, fully tested and verified, replace all existing hardware that used the old ones by either reworking the assemblies or building new ones that have the new parts.....
I'm guessing Space-X will opt for the latter in public, but unless they sue the supplier for damages, the problem really is the former, which scares me..
Re: (Score:2)
So uh... "Further testing of struts in stock found one that failed at 2,000 pounds of force"
Sounds to me like they likely were testing them at random and then decided to start testing a vastly significant number of them to troubleshoot.
Re: (Score:3, Insightful)
So uh... "Further testing of struts in stock found one that failed at 2,000 pounds of force"
Sounds to me like they likely were testing them at random and then decided to start testing a vastly significant number of them to troubleshoot.
Which tells me they may have a QA process issue that compounds a design issue. NEVER do you take a single failure risk in a human rated system's structural components unless you are SURE the components are sound. So if this strut was a critical structural component without any backup and you are not totally sure it can take the load before you fly the assembly, it's a process problem.
They do want to make this a human rated system right? That goal is in serious jeopardy now. You cannot afford to "learn
Would have saved itself (Score:5, Informative)
Pilot wouldn't have needed to. Dragon 2 has automatic abort capabilities (even when unmanned). It would have separated from the second stage - probably firing its SuperDraco thrusters - and then automatically deployed parachutes once it was a safe distance away.
Dragon 1 doesn't have the SuperDracos (only the much smaller Draco attitude control thrusters) so it wouldn't have been able to put as much distance between itself and the booster, but from the video and the telemetry it looks like the capsule survived the (accidental) separation anyhow. It could have deployed its parachutes and probably survived the landing, but it wasn't programmed to do so. They have added it to the Dragon 1 programming now though.
Failures that occur high enough to land under parachutes, slow enough to get away from the inevitable explosion without heavy rockets, and early enough in flight that there's no time to manually enable the landing sequence are... really, really rare in rocketry. Usually you either fail at liftoff (see Orbital's last attempt to launch Antares), fail rapidly and catastrophically during liftoff (any number of examples), or fail once in orbit (often, though not always, at stage separation). In orbit you have time to make a decision and send orders. On the launchpad you can't land safely (without abort rockets). In midair you *usually* can't get away in time (without abort rockets). This was an exception to the "in midair" usual failure case; there were nine seconds from beginning of the failure to loss of vehicle, and in fact the capsule had already tumbled free (and probably *could* have used its ACS thrusters to put some extra distance between itself and the booster.
One thought, though: what about, in the case of a pre-separation second-stage failure, executing MECO 1 (Main Engine Cut Off, when the Falcon 9 first stage kills its rockets) early and doing an emergency stage separation? Normally there's no point - the first stage on most launch vehicles has no purpose if the launch fails and nowhere to go even if it separates safely - but the Falcon 9 first stage is designed for reusability. Emergency MECO, separate the stages, use the ACS and/or grid fins to steer clear of the second stage, and then fire up the main engines again and aim for the droneship or other landing pad. You'd need to be quick about it, and it might still not work, but if it does you've saved a booster worth $70,000,000 USD. Well, that and demonstrated the first successful first stage recovery ever, but assuming that becomes as routine as Musk wants it to be...
Actually, it would have been super cool if the first successful recovery of the first stage had been an emergency abort!
Re: (Score:3)
Actually, it would have been super cool if the first successful recovery of the first stage had been an emergency abort!
Something like this actually happened on the fourth or fifth flight of the DC-X test vehicle. It had made several successful launches, hovers, and vertical landings and the test profile had it going to higher and higher altitudes and doing some interesting maneuvers. Anyway, on the nearly fatal flight, due to wind conditions in the launch area, hydrogen gas (it was LH2/LO2 fueled) coll
Re: (Score:3)
Random testing can only pick up systemic faults within an entire product line not random ones. You have to try and ensure the random samples are as representative of the collective as possible which can be helped by doing things such as random testing at least one from every heat number but ultimately you will never catch them all.
Based on the summary itself:
Further testing of struts in stock found one that failed at 2,000 pounds of force
emphasis mine. This implies that testing other ones would have shown no problem unless it was possible to classify all the failed ones by some common b
Re: (Score:2)
Random testing can only pick up systemic faults within an entire product line not random ones.
Statistics is your friend in this case. Random testing should show a large standard deviation (assuming they test to failure). You should then be able to calculate the probability of failure.
...Or some shlub forgot to heat treat two of the struts.
This may not be so random.
Re: (Score:3, Insightful)
Statistics is your friend in this case. Random testing should show a large standard deviation (assuming they test to failure). You should then be able to calculate the probability of failure.
No, just no. Statistics helps if you have a process variation that approximately follows a normal distribution. It does nothing to protect you against freak failure, like say you're making creme brulee [wikipedia.org] using a torch to caramelize the sugar on top but the spark to light it is only 99.9% reliable. That means 999 out of 1000 desserts will be fine and one will be a total failure, but you don't know it until it happens. Failing at 1/5th the design load is clearly outside any normal variation and tells you none o
Re: (Score:2)
Statistics may be your friend but cash is your enemy. It's hard to justify the cost of destructively testing more than one thing in any order, even harder if most of them come back without issue from a preferred vendor who's never done you wrong.
Re: (Score:2)
That is why you use Statistical Process Control. I used to build assembly equipment and many tolerances were controlled by statistics. For eample say you had a certain diameter on a part that needed to be within tolerances. The first thing you do is take data on random parts to make sure all you have are random variations. If there is anything non-random you need to fix it. Then you set your control limits such that your process with your required degree of certainty is within it. So if you need the part to
Re: (Score:2)
That is why you use Statistical Process Control.
I'm familiar with the system, but am still glad they do ultrasonic inspections of stress-critical airplane parts, not just a statistical sampling.
This has to do with costs vs. expenses. Settling all those wrongful death suits is expensive. Buying new rockets and payloads is expensive. Replacing dead cell phones is cheap.
If I were Musk, I'd hire some techs to do ultrasonic inspections of every strut on the way in. Or, even better, set a contract so that th
Re: (Score:2)
It's not just about the cost of a failed launch, there's also a huge cost to a company's reputation if a rocket fails. And to their schedule.
Out of curiosity, is there any lightweight way to sense how close a part is to failure *in use*? I mean, finding defects on the ground is great, no question. But what if something would doom a mission not due to a part having a manufacturing defect, but due to an oversight somewhere in the rocket design process, or assembly, or transportation, or launch setup, or unexp
Re: (Score:2)
To pick a random, for example, would there potentially be a change in resistance or capacitance or other electrical properties when a strut nears its breaking point?
That ought to be easy to test, at least experimentally, using capsense...
Re: (Score:2)
That is a great thing to incorporate by the vendor, not so great when you're buying a product. As an end user for a product someone else supplies it gets hard to justify the cost of destructive testing, sometimes it's even hard to justify the cost of non-destructive testing of items bought from other people.
Re: (Score:2)
That said, it is pretty stupid that Space X has not been testing random parts to confirm they meet the requested specifications.
From the article:
That led the company to test what Elon Musk described as an “enormous” number of struts, where they found another strut that failed under the same conditions.
How many struts would they a have to randomly test to catch the two out of an "enormous" amount that actually failed? And do you honestly think they aren't already testing parts?
But, clearly you know better than they do, so here, go show em [spacex.com]. Maybe there is "strut tester" in there somewhere.
Re: (Score:2)
I know that no part has a 0% defect rate, so perhaps they should engineer their system so that a failure of no one strut is not catastrophic.
If the strut would need to bear 2000 pounds on its own, then they should spec. it to the manufacturer for 4000 pounds and add more struts to spread the load.
Re: Transparency (Score:2)
FTFS: ". The failure occurred at 2,000 pounds of force, and the struts were rated at 10,000 pounds of force". We don't know what the maximum force expected is.
As for more struts, when you look at the Soyuz interstage lattice, I wonder what would happen if just one those failed...
Re: (Score:3)
Outside of classified launches... when exactly was the government not this transparent (and more)?
$10,000 toilet seats (Score:3)
Re:$10,000 toilet seats (Score:5, Insightful)
Yep. Once you have thousands of parts all designed to trade off as much strength for weight reduction as possible it doesn't take much of a manufacturing hiccup to cause an expensive "excursion". Vendors end up having to rigorously test every widget, and custom design it just for you.
Before long $10k each for a batch of a half dozen toilets seat that are space rated to not outgas funny chemicals that foul optics, handles 10g's, has 6 sigma of de-rating for the bolt hole strength, weighs under 500g, and is non-flammable starts sounding like a deal.
Re: (Score:2)
Before long $10k each for a batch of a half dozen toilets seat that are space rated
There is a justification for this however.. If you are sitting on the only potty in HQ and the toilet seat collapses beneath you, then your ass is in deep shit....
Re: (Score:2)
The supplier who gave them bad or subgrade steel should be, they're about to get reamed...
Bob Seger: (Score:2)
"They love to watch her strut!"
Re: (Score:2, Informative)
I think you hit the nail on the head with that $250 hammer...
Space X has apparently having systemic Quality Control problems. A structural failure of ANY component that costs you a vehicle in flight is a SERIOUS issue with your process. Either they didn't recognize this component's critical nature in their design reviews and properly established safety margins for it's strength, or they unknowingly used substandard parts in the assembly.
Structural failure should NEVER be the cause of an accident. Str
Re:certified materials (Score:4)
You think having the part designed to handle five times the load it actually experienced to not be "with sufficient margin"? How much of a margin do you want them to put, 100x?
RTFA. They were doing statistical-sampling quality control testing of struts. The problem was that most of them were just fine, but there were a very small number which were totally defective and broke at a tiny fraction of their rated value. And no, SpaceX did not make the parts, it was an outside supplier. And yes, SpaceX A) will now be testing 100% of them, and B) is ditching the supplier.
Re: (Score:2)
TESTING requires destruction of this kind of thing as I read the article. They will NOT be doing 100% testing to failure of their stock of struts, except to prove to themselves how bad their supplier really was.
Apparently their testing process was insufficient to be sure, which is my point. You DON'T put a flight critical structural component into an assembly unless you are SURE it's going to work at least to the design limits required. Structural failure is a serious problem in the aerospace world and b
Re: (Score:3)
TESTING requires destruction of this kind of thing as I read the article. They will NOT be doing 100% testing to failure of their stock of struts, except to prove to themselves how bad their supplier really was.
Nobody said they would be testing to failure. You can test every unit to say 150% of failure. If the material is rated for 1000% of failure then 150% should be safe. If it doesn't fail at 150% once it probably won't fail at 100% 100 times. So now you're at 99 times until mean failure instead of 100.
Re: (Score:2)
Re: (Score:2)
Shut the fuck up, nameless twit
Re: (Score:2)
way to go.
Re: (Score:2)
Re: (Score:2)
After the doge meme's 15 minutes of internet fame, your sentence actually much parses.