How a Tiny Bit of Lacquer Grounded New Falcon 9 Rockets For a Month (arstechnica.com) 21
On October 2, SpaceX automatically aborted the launch of a Falcon 9 rocket (with just two seconds left in the countdown) after two of its first-stage engines ignited early. Investigating the problem ended up delaying the October 31st launch of four astronauts to the International Space Station until November 14, reports Ars Technica, citing a teleconference with SpaceX's Hans Koenigsmann.
"Had the abort not triggered, it is likely that nothing bad would have occurred, but Koenigsmann said that under certain extreme scenarios, rattling from an early ignition may cause significant damage to the Merlin engines." SpaceX technicians removed the two engines and shipped them from Florida to the company's test site in McGregor, Texas, where they were able to replicate the problem. They found that a relief valve within the gas generator — a tiny rocket within the engine that starts up and powers its machinery — was clogged with a masking lacquer akin to nail polish. They were able to show that removing the lacquer from the vent hole allowed the engines to start up normally. This lacquer is applied during an anodizing process to treat aluminum components of the gas generator. It is supposed to be subsequently removed, but in the case of these two engines, a tiny amount of the material had been trapped within a bore hole less than 2mm across...
After this, SpaceX inspected other engines across its fleet (the company inspected new boosters only, as Falcon 9 first stages that have already flown are not subject to this issue). SpaceX found that two of the engines on the Falcon 9 rocket that will be used for the Crew-1 launch also had this problem. Those two engines are now being swapped out for new Merlins...
This lacquer issue came as a surprise given that SpaceX has literally launched hundreds of new Merlin 1D rocket engines over the last decade and, until now, had not seen this problem with the masking agent. "It's certainly possible that we had cases of it earlier, and they were appreciably so harmless that we completely missed them," Koenigsmann said. It is also possible that a small process was changed so that all of the lacquer was not removed, as this particular treatment is done by an outside vendor. But, he admitted, "It's difficult to explain how this works for so many years and then, suddenly, you see this coming up in the data."
What Koenigsmann seems confident in is that the issue will not occur again.
"Had the abort not triggered, it is likely that nothing bad would have occurred, but Koenigsmann said that under certain extreme scenarios, rattling from an early ignition may cause significant damage to the Merlin engines." SpaceX technicians removed the two engines and shipped them from Florida to the company's test site in McGregor, Texas, where they were able to replicate the problem. They found that a relief valve within the gas generator — a tiny rocket within the engine that starts up and powers its machinery — was clogged with a masking lacquer akin to nail polish. They were able to show that removing the lacquer from the vent hole allowed the engines to start up normally. This lacquer is applied during an anodizing process to treat aluminum components of the gas generator. It is supposed to be subsequently removed, but in the case of these two engines, a tiny amount of the material had been trapped within a bore hole less than 2mm across...
After this, SpaceX inspected other engines across its fleet (the company inspected new boosters only, as Falcon 9 first stages that have already flown are not subject to this issue). SpaceX found that two of the engines on the Falcon 9 rocket that will be used for the Crew-1 launch also had this problem. Those two engines are now being swapped out for new Merlins...
This lacquer issue came as a surprise given that SpaceX has literally launched hundreds of new Merlin 1D rocket engines over the last decade and, until now, had not seen this problem with the masking agent. "It's certainly possible that we had cases of it earlier, and they were appreciably so harmless that we completely missed them," Koenigsmann said. It is also possible that a small process was changed so that all of the lacquer was not removed, as this particular treatment is done by an outside vendor. But, he admitted, "It's difficult to explain how this works for so many years and then, suddenly, you see this coming up in the data."
What Koenigsmann seems confident in is that the issue will not occur again.
Not unexpected mid-program issues (Score:5, Interesting)
I haven't been tracking Falcon launch delays, but it seems like there have been more than the usual technical issues.
I was expecting them to be due to the multiple launches the boosters that have been used (especially for Starlink) would reveal some longer-term reliability issues with some parts that don't work as long as projected. I think there are a couple of boosters that have flown five times and maybe one that's flown six so I would expect some things to break, even though they weren't designed to.
I thought the quote from Koenigsmann "It's difficult to explain how this works for so many years and then, suddenly, you see this coming up in the data." was interesting because it could be they've had this problem all along, but now with a significant amount of data (they just did 100 successful launches) they should have a good feel for how the engines run and what is an anomaly. I'm sure that in the start up and main ignition of large rocket engines, a "rattle" like this is hard to detect, but with lots of data that allows tuning of parameters, it starts to become obvious.
Re: Not unexpected mid-program issues (Score:2)
Re: (Score:2)
they've had this problem all along, but now with a significant amount of data
I don't think so. It's more a case of some parameter going over a threshold and triggering a shutdown. Not based on the collection of 100 flights of statistical data.
The second assumption is more likely:
It is also possible that a small process was changed so that all of the lacquer was not removed,
The person that cleaned the nozzles probably retired and an (undocumented) cleaning step was not followed by their replacement.
Re: (Score:1)
I thought the quote from Koenigsmann "It's difficult to explain how this works for so many years and then, suddenly, you see this coming up in the data." was interesting because it could be they've had this problem all along, but now with a significant amount of data (they just did 100 successful launches) they should have a good feel for how the engines run and what is an anomaly. I'm sure that in the start up and main ignition of large rocket engines, a "rattle" like this is hard to detect, but with lots of data that allows tuning of parameters, it starts to become obvious.
Alternate explanation: 3rd party contractor got sloppy.
This is one example... (Score:5, Interesting)
There's a reason people equate rocket science with really difficult things to do.
What amazes me is not just the intricate detail of these rockets that a thin layer of lacquer grounded them but also that they were even able to detect the flaw. It seems that they were not as concerned about safety as much as premature wear on the engines. They stated that it would take rather extreme circumstance to damage the engines had this not been fixed when it had. Given that these rockets are built with many redundancies the loss of an engine during launch would still allow for mission success. Even then SpaceX has experienced a loss of a rocket and the mission was still considered a success. This is because recovery of the rocket is merely a bonus, the goal is getting stuff into space. It's expensive to lose a rocket, and so that explains why they'd want to take the time to track down this issue.
Another thing this shows is that the more experience people gain the easier it becomes to succeed.
Re: (Score:2)
Re:This is one example... (Score:5, Informative)
They literally had to CT scan the gas generators to find it.
This reminds me of something other than rockets (Score:5, Interesting)
This reminds me of a story I heard about the difficulty of documenting and duplicating processes in an unrelated field. It was some kind of food production facility that had to be moved. We're talking about a large company with a massive plant, and the possibility that customers would be displeased if the product didn't have the same taste and texture. So they documented everything about the process, procedures, and inputs before moving; or so they thought.
When the first run went through, the taste was noticeably off. A huge multi-million dollar endeavor was at stake, possibly the company itself if they couldn't figure out the problem. No, this isn't "mission critical" in the same sense that a manned rocket is; but it was certainly important for the many people connected to the company, employees, etc.
Ultimately they solved the problem by going back and interviewing workers. They discovered an undocumented routine that involved carrying ingredients from one part of the plant to another. Apparently the temperature of those ingredients was important, the cooling while carrying was critical, and they had never been aware of it. The plant had operated for decades prior based on "this is how we do it".
Re: (Score:2)
There are many stories like this. That is why "source code control software" was invented.
Imagine the first several Saturn V rockets. They all where slightly different and changes were not well documented. It was basically in the mind of the engineers or spread over document in different engineering offices.
There is a story about a chip plant, that was running exceptionally good. So when they wanted to build a new one they copied that one. The original had a problem during construction, so some pipes transp
Re: (Score:2)
The semicon industry (Intel in particular) has a program that they enforce with vendors to give validity to that story about the chip plant. They call it "Copy Exact!" (the exclamation mark _is_ part of the name)
They would rather simply hold everyone in the supply-chain accountable for any trivial change in process or material rather than characterize exactly every part of their process and test all of those parameters every time input product or equipment is received. I get the point of it, but at some
Re: (Score:2)
Interesting.
Nice nick btw.!
Laudable transparency (Score:5, Interesting)
I am impressed with the level of transparency they are exhibiting with their investigation, but not only that their candor with what they are thinking. One thing you will never see NASA do is exhibit uncertainty. The first thing that happens with a problem is the outside communications lines are cut. The second thing is that engineers and decision makers get locked in and the only people who speak are PR peeps.
I have always been wary of the privatization of space - the corporatization rather. And I still am wary with the ultimate end goals of SpaceX et al, but they are a group who is doing it right. They are doing it all right.
Like most everyone else, I had a love affair with NASA in the 80s with the shuttle and Voyager and all the rest. I resisted disillusionment with the revelations after Challenger. But then came the Mars Climate Orbiter, Columbia, Curiosity, SLS. Debacle after debacle. Which makes you realize it's endemic in NASA's trough-eating self-protective pork barrel mentality, and so you go back and revisit the love affair you had with the Shuttle and realize that was all smoke and mirrors. The shuttle was a terrible program, and the whole infrastructure around it was designed to mask how truly awful it was. It was not a success - they lost 40% of their fleet and 4% of their crew in preventable circumstances, and both investigations resulted in findings of poor decision making and both said the problems at NASA were systemic. And now we are giving them more billions that every private spacecraft builder combined for SLS, which is years late, triple the cost, and at best will provide no new technology because it's actually just reusing the worst parts of the already terrible shuttle program.
So I applaud SpaceX's transparency and openness here. It is something I wish NASA had ever demonstrated. Maybe if they had been as transparent, a lot of the pain could have been avoided. I wish we could avoid the future pain that is and will be SLS too, but someone still insists on giving NASA money.
Re: (Score:3)
Well,
the sad thing is until the first accident, Challenger, right? I did not even know that the crew rescue vehicle was scratched from the design. (I read a lot about the upcoming Space Shuttles when I was a boy, saw the first launch on TV, obviously)
And the horrible thing is: the crew survived the explosion, and died on impact into the sea. Had the rescue vehicle, basically the whole cockpit would have been separated and going down on parachutes, not been canceled by "been counters, oh it is so expensive!!
Re: (Score:2)
Colombia, disintegrated during reentry. Some smart people suggested a different entry angle, that would have given the vehicle and the crew a quite high chance of survival: NASA rejected it. Of course we do not know if it had worked ...
This is the real issue:
"In a risk-management situation similar to that of the Challenger disaster, NASA management failed to recognize the relevance of engineering concerns for safety and suggestions for imaging to inspect possible damage, and failed to respond to engineers' requests about the status of astronaut inspection of the left wing. Engineers made three separate requests for Department of Defense (DoD) imaging of the shuttle in orbit to determine damage more precisely. While the images were not gua
Re: Laudable transparency (Score:1)
Re: (Score:2)
Wow, never heard about that one!
Playing with other peoples live in an "civilian organization" - absurd.
Space shuttles. (Score:4, Insightful)
SpaceX is handling this correctly.
NASA leading up to 1986 argued: It few many times with an eroded O-ring, so it can't be that bad.
NASA leading up to 2003 argued: We've had foam blocks hit the shuttle before, so it can't be that bad.
SpaceX has the software configured to abort on "odd measurements" of the system.
They they investigate, find the cause, and fix it even if "nothing bad would've happened". F*** launch-pressure. STOP, investigate, fix.