NASA Safety Panel Calls For Reviews After Second Starliner Software Problem (spacenews.com) 83
A second software problem during a CST-100 Starliner test flight is prompting a NASA safety panel to recommend a review of Boeing's software verification processes. Space News reports: That new software problem, not previously discussed by NASA or Boeing, was discussed during a Feb. 6 meeting of NASA's Aerospace Safety Advisory Panel that examined the December uncrewed test flight of Starliner that was cut short by a timer error. That anomaly was discovered during ground testing while the spacecraft was in orbit, panel member Paul Hill said. "While this anomaly was corrected in flight, if it had gone uncorrected, it would have led to erroneous thruster firings and uncontrolled motion during [service module] separation for deorbit, with the potential for a catastrophic spacecraft failure," he said.
The exact cause of the failure remains under investigation by Boeing and NASA, who are also still examining the timer failure previously reported. Those problems, Hill said, suggested broader issues with how Boeing develops and tests the software used by the spacecraft. "The panel has a larger concern with the rigor of Boeing's verification processes," he said. The panel called for reviews of Boeing's flight software integration and testing processes. "Further, with confidence at risk for a spacecraft that is intended to carry humans in space, the panel recommends an even broader Boeing assessment of, and corrective actions in, Boeing's [systems engineering and integration] processes and verification testing." The panel added that all those investigations and reviews be completed as "required input for a formal NASA review to determine flight readiness for either another uncrewed flight test or proceeding directly to a crewed test flight."
The exact cause of the failure remains under investigation by Boeing and NASA, who are also still examining the timer failure previously reported. Those problems, Hill said, suggested broader issues with how Boeing develops and tests the software used by the spacecraft. "The panel has a larger concern with the rigor of Boeing's verification processes," he said. The panel called for reviews of Boeing's flight software integration and testing processes. "Further, with confidence at risk for a spacecraft that is intended to carry humans in space, the panel recommends an even broader Boeing assessment of, and corrective actions in, Boeing's [systems engineering and integration] processes and verification testing." The panel added that all those investigations and reviews be completed as "required input for a formal NASA review to determine flight readiness for either another uncrewed flight test or proceeding directly to a crewed test flight."
Who'd a thunk it? (Score:5, Funny)
Just wait till somebody tells NASA about the 737 MAX!
Re:Who'd a thunk it? (Score:5, Insightful)
Funny how few people are still complaining about SpaceX being irresponsible and cutting corners with safety. They did their in flight abort test which wasn't even a NASA requirement, while Boeing decided that a software simulation would do just fine. And now Boeing's even trying to avoid even having to redo a failed ISS rendez-vous test flight. No big deal, it's just some software bugs. If that kind of minor bug could kill, airplanes would be falling out of the sk... I mean, trust us, it will be fine.
Who's cutting corners now?
Re: (Score:2)
Re:Who'd a thunk it? (Score:5, Insightful)
Re, the valve failure - the emergency abort system in practice should never be used. For the problem that occurred to actually destroy a Dragon in practice, you would have had to perform an emergency abort a minimum of twice in the same craft - once for the defective check valve to not close all the way, and a second for it to cause the blowback problem with the residual propellant. And have not noticed signs of the failure in the intervening time (vs. in testing where there's relatively little downtime between tests).
The problem was only discovered because SpaceX repeatedly tested their abort system both as on-ground abort, in-flight abort, and numerous hold-down aborts. By contrast, Boeing tested their abort system only once. From the ground only. And during the test, had a parachute failure and landed the craft in a plume of toxic gas (the latter, by design...).
I say this not to excuse SpaceX, BTW (good they found it and fixed it), but rather, as criticism of Boeing's lax testing programme. These aren't cargo flights - there's going to be humans onboard these things. You don't cut corners. We don't need a "Starliner MAX" here. :P
Re: (Score:2)
I say this not to excuse SpaceX, BTW (good they found it and fixed it), but rather, as criticism of Boeing's lax testing programme. These aren't cargo flights - there's going to be humans onboard these things. You don't cut corners. We don't need a "Starliner MAX" here. :P
IMO, there's nothing that requires excusing in SpaceX's case. It's a new bit of a rocket, and there are almost guaranteed to be some design issues. The thing is, SpaceX seems to have a culture of genuinely trying to do it right (as one of the ancestor posts pointed out, they're doing tests that aren't required by their contract), and when it goes wrong, once they figure out what happened, they have no problem with, "hey, we f--d up, here's how, and we're fixing it for next time."
Re: (Score:2)
While I am not fan on Elon, I do admire his teams, They really do encompass the desire to
achieve and be known as "Right Stuff", testing and testing and test, knowing that a buddy
of theirs might be on that rocketship
They practice what my father taught me as a kid, design it, then try to break it, if it breaks where
it's suppose to break then fine, if not, fix and re-test ( we would work on cars all the time, shaving
weight, testing idea's, fun weird shit as a kid, we even copied Smokey Yunick propeller driven
a
Re:Who'd a thunk it? (Score:4, Insightful)
Rocket science is hard. You test, test, and retest, and then if you don't find anything wrong you test again. Any problems get brought back to the drawing board, then repeat.
Boeing claimed its processes were so good they could "test on paper" and then fly a qualifying flight. Everybody else expected this result.
I suspect all the old engineers of yesteryear got retirement packages from the beancounters. Actually there are some notable industry veterans at SpaceX. Maybe I'm wrong and it's not all kids and middle managers skating by on company reputation at Boeing.
Re: (Score:2)
"Starliner MAX": I think you've discovered a new meme. Well done!
The triumph of Crapitalism!(tm) (Score:5, Insightful)
Diversity hires, FTW!
Well, if you consider adding MBAs to a Engineering driven endeavor to be "diversity hires".
But from what I read, it was the McDonnell Douglas management driving out the Boeing upper managers that led to the catastrophic culture shift.
But hey, they created a lot of value for shareholders [twitter.com] until the whole rotten edifice collapsed around their ears.
Not a great start (Score:4, Insightful)
---
Re: (Score:2)
This will be the third black eye after the previous satellite launch and 737-Max problems.
Are we not going to talk about the fact that Boeing has three eyes?
History (Score:5, Interesting)
Perhaps Boeing's management and programmers could take a lesson from Don Eyles, Margaret Hamilton, Katherine Johnson, et al.
It's rocket science and way beyond me, but we sent three men to the moon 7 times using reams of assembler stuffed into core rope memory.
Re: History (Score:3)
Re: History (Score:5, Interesting)
Have you seen this great video on the Apollo program's memory modules?
https://www.youtube.com/watch?... [youtube.com]
Re: History (Score:2)
Re: (Score:2)
That too was not without it's small share of issues:
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
Rocket science is hard because of the engines. Once you've got the engines the rest is much, much easier. Boeing is currently using Russian engines, but Starliner should be able to use SpaceX ones too.
Re: (Score:2)
The SpaceX ones are descendants of copies of Russian engines. Sergei Korolev should be as well-known as Von Braun, but all during the Soviet times he was just known as Chief Designer.
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
The Merlins aren't related to Russian engines, they're pintle-injector gas-generator engines that originally had ablative nozzles and combustion chambers. They're more or less descendants of the FASTRAC engine (which never flew).
The Raptor isn't closely related to much of anything, being the first FFSC engine to fly, though its oxygen-rich preburner might owe some heritage to Russian staged combustion engines.
Re:History (Score:4, Interesting)
You idea presumes that assembly code and core rope memory (and only small amounts of it, at that) are some sort of significant limitation.
It isn't - both are far more suited to the task than what most people here would recognize as "modern programming practices" in faster computers. If you don't have modern computers, you also can't have modern programmers, most of whom know how to write code fast, and when possible, to not write it at all. It's geared towards development speed, not necessarily toward being more correct. If you talked to a lot of people, they don't know and haven't even heard of the concept of "correctness" the way it was used in the past for life-critical embedded processor code. And modern software testing practices are completely unsuitable.
There seems to be a lot of things that have led to this, but underlying it all is that really fast computers just enable complexity, in fact, require it, and complexity is the mortal enemy of correctness and why it is essentially impossible to test it to completion. The fact that almost everyone reading this is rushing to their keyboards to argue "what do you mean, test to completion? That's a ridiculous idea, you can never completely test anything!" is precisely what I mean.
The biggest impediment to space flight today, and return to the Moon and on to Mars, is *software development and the widespread availability of really fast computers*. Those are not really positives - computing power was completely adequate for Apollo at the time, it was no significant impediment on either the spacecraft or on the ground, and the lack of computing power made them *more likely to succeed*. The last important development in spaceflight computer design was the widespread availability of floating-point machines, and the last important development in software was the development of suitable purpose-designed high-level languages like JOVIAL (early 60s) and FORTRAN 77+the usual extensions (1977). After that, no matter now much "better" the computers got, no matter how much more sophisticated the processes got, they did not improve the ability to develop or test the sort of software required for this sort of application - quite the opposite.
Re: (Score:2)
Sadly, many people are not inclined to learn even the basics of how to code cleanly,
with decent documentation. Once you got a team the can code in similar structures,
it no longer a chore to do code cleanup and sectional rewrites.
I saw two old print out of my fortran and basic code, you could read everything and
duplicate the code in any language with the notes. I kinda miss those days when you
coded to get quality results the fastest way. My AD&D lookup basic program was
freaking amazing now that I look ba
Re: (Score:2)
This is basically what I tell people when they suggest that Star Trek's constantly exploding computers are unrealistic. Extrapolating from the current trend of increasing complexity in computer systems, it's a miracle that the Enterprise doesn't literally catch on fire the moment they turn the key in the ignition.
Re: (Score:2)
If you talked to a lot of people, they don't know and haven't even heard of the concept of "correctness" the way it was used in the past for life-critical embedded processor code. And modern software testing practices are completely unsuitable.
Ok boomer. Meanwhile on Apollo...
https://www.wired.com/story/ap... [wired.com]
The alarm subsided, but just seconds later came another reboot, another dropout of the display, this last one just 800 feet or so above the surface. That made five crashes in four minutes, but the go commands from Houston kept coming.
Re: (Score:2)
Wired, really?
The code did what it was supposed to do - the CPU was overloaded with spurious information from an input that wasn't strictly necessary at the time (rendezvous radar), so the executive saved the job state of critical tasks, flushed the queue of non-critical tasks and restarted from where it left off. It didn't crash or reboot. The critical functions continued to operate and the descent and landing completed without further incident.
There's a much better and more detailed explanation at https:/ [nasa.gov]
Re: (Score:2)
Script-kiddies should be seen and not heard.
If you had done even a milliseconds worth of actual research, you would find that there were no *reboots*, there were no real impediments to the mission, and, as they determined - in the space of 30 seconds or so - that these were alarms that had no bearing and the mission could continue, which it did, successfully. The issue was not a problem with the code, but, arguably and to simplify it for the immature mind, a misconfiguration of the switches that resulted i
If I was an astronaut (Score:5, Insightful)
I'd be checking out my retirement options, just in case Boeing manages to bribe its way into, er I mean win, the final contract.
Re:If I was an astronaut (Score:5, Insightful)
Re: (Score:2)
They've already won the contract along with SpaceX, this is them "delivering".
And if I was an astronaut, I'd probably be getting together with the others and developing a "we all refuse to fly on this, unless it passes these actual test (not software simulations)" type of letter to NASA.
Re: (Score:2)
Keep in mind that when the original Gemini 7 were brought to a launch pad to see the rocket they were going to fly in take off it blew up on the pad in front of them. None of them quit.
Probably more likely would be a letter from the families of the astronauts to NASA.
Re: (Score:2)
You meant Mercury 7.
It was an Atlas test, and Al Shepard's reaction was classic: âoeIâ(TM)m glad they got that one out of the way. I sure hope they fix that.â
Re: (Score:2)
...and this is after we gave them extra money. [geekwire.com]
Here's the problem ... (Score:5, Funny)
The panel called for reviews of Boeing's flight software integration and testing processes.
The original name for the Boeing capsule was Starliner MAX, so ...
Re: (Score:1)
It has nothing to do with the origin of the programmers - be they from the US, India, China, or Ebonia. It has to do with the rigor an maturity of their development program. When you take all the folks that know what they are doing and "retire" them to save costs you get 737MAX. When you put greenies into rocket science you get Starliner.
For anything dealing with life-systems and/or Aerospace the teams should be at a level 4 on the CMM scale.
Level 5 would be ideal. (https://en.wikipedia.org/wiki/
Re: (Score:2)
This isn't a worker problem. It's a management/organizational one.
Modern Corporate America "leadership" in a nutshell. I was going to say Boeing, but the problem extends through our entire economy today.
Re: (Score:2)
but the problem extends through our entire economy today. ...
And politics
Is it too late to short Boeing? (Score:3)
Because regardless of Tesla stock values, Boeing seems to be cutting their own throats lately...
Re: (Score:2)
shorting is fundamentally flawed. Let's watch what happens if you try to sell your landlord's house.
Re: (Score:2)
Re: (Score:2)
What's wrong is that theoretically you justify your bid by accepting unlimited risk, but in actuality if that happens you declare bankruptcy and put a hard limit on the risk.
Re: (Score:2)
By the time news hits mainstream media it is generally to late to short something, sure. As the stock is up since the start of the year, it may actually be a good moment for when something else hits? Who knows...
Paying more for this... (Score:2, Insightful)
And NASA is paying Boeing more compared to SpaceX for this crap. When will they realize the Boeing of the 90s, 80s and back is no more and hit them so hard it actually makes a difference?
Re: (Score:2)
When Boeing stops offering board memberships to retiring congresscritters and generals.
(Or maybe when a Boeing board membership is no longer worth time spent in the monthly conference call.)
Boeing is dying (Score:5, Insightful)
It seems today the only way that Boeing can succeed is by performing their own safety inspections instead of the FAA, by bribing legislators to make them the chosen company for space travel, and by promising that their failures are one-offs that can be fixed without testing and without oversight from anyone else. At this moment, the whole company appears doomed.
Even if the US tries to prop up Boeing, the reality at this point is that Airbus and SpaceX have taken the high ground in two major areas in which Boeing competes, and Boeing is absolutely going to fail if they don't revolutionize the whole company. They need to toss the greedy stooges off the board and replace them with people who value greatness and vision, they need to dump the MBAs and replace them with engineers, they need to return to a culture of perfection and safety and stop hiring low-wage middle management and assemblers who take any shortcut to get the plane off the assembly line.
They're ruining the company, and they're watching it circle the drain, and they don't seem to know why it's happening. But the answer is easy. They stopped putting quality first, and nobody in charge knows how to go back to that.
Re: (Score:2)
Re: (Score:2)
space age mentality (Score:2)
Re: (Score:2)
I think that is exactly what they did - and this is the result.
Crazy (Score:4, Interesting)
Re: (Score:3)
To be fair, sometimes software can fix bad design. But it takes really good software, and that's not easy either. (And, of course, often it can't.)
Re: (Score:2)
What's the 'lesson' for the managers though? The ones that first took over and asset stripped the company have already banked the gains and are living it up on the proceeds. Those who are in charge of the current fiasco are not going to have to pay back the millions they have extracted from the business, and will all get golden parachutes if they have to move aside. The management team that comes in to 'fix the mess' will probably be paid even more money, and have no responsibility (they will just say the p
Re: (Score:2)
The lesson is for the shareholders, who appoint the managers, and who are currently taking a bath. Usually it works out exactly as you outlined, but in Boeing's case I think maybe there will be an actual lesson.
Re: (Score:1)
Shareholders don't appoint the managers.
Re: (Score:2)
Right. The CEO does. And Calhoun (be he a good guy or not) is only in that seat for a few years. Not long enough to root out all the idiot son-in-laws of the company execs.
Re: (Score:2)
Sure they do. Shareholders elect the board (managers) who appoint the CEO (manager) who appoints more managers, who appoint more managers, etc. Sometimes shareholders directly approve the appointment of senior managers like the CEO too. Let's see, I think I have a voting form around here somewhere... yup, "blah blah confirm X as CEO blah blah."
What dreamliners may come. (Score:1)
So, basically... (Score:2)
...Boeing and the Iowa Democrats used the same software developers.
Hear Ye! Hear Ye! Bring out your conspiracy theory (Score:1)
Re: (Score:2)
That would be a massive risk for both Boeing and NASA. If they lost a crew....
If I were in charge of Boeing and NASA made that offer I would politely decline. Maybe not politely.
Re: (Score:2)
Re: (Score:2)
at their own cost
Time for the DoD to order a few toilet seats and a box of hammers.
Perfectly safe (Score:2)
The Boeing 737 Starliner is perfectly safe. It's the best spacecraft ever produced, it's perfect. It wasn't the software, it was pilot error.
Fuck programmers (Score:1)
Re: (Score:2)
Software scales more easily. Doctors can only kill one a a time.
Re: (Score:2)
Someone made mention of it before, it's called coding to correctness.
Back in the day, you would spend time placing the code concepts together
then write out the code outlines ( we still do it, but no longer on index cards or white boards )
then as a team, you would decair your steps, variables, whatever you were usings
and how you would close them, give back the memory, and decommission the variable declarations
then it was coffee, all nighters, and yelling at someone's crappy non documented code.
takes 10 times
Re: (Score:2)
1. Requirements
2. Flowcharts
3. Pseudo-code
3a Define testing requirements
4. Coding standards
4a Write test routines/procedures and outcomes
5. Code
6. Compile
7. Testing round 1
8. Repeat 5, 6, 7 until no more errors, and all test outcomes are met.
Re: (Score:2)
Quick question :
even if all test outcome come correct,
how do you test for variable problems that come out of scope...
Reason I ask is the following: just recently ( within the last 5 years )
I reviewed a piece of equipment after a failure. I found that the
failure occured when the inputs were beyond the extreme known
( it was a specific maritime navigational system failure, think ship
throttling over a wave at a 20' angle with pitch and roll of what it would
like in a hurricane ) and it did not hand over the cont
Re: (Score:2)
The test routines would include testing for input "out of scope", e.g.if the range of acceptable inputs is 50 - 400, the tests would include values within that range, but also a negative number, 10, 49.999 (if the input allowed* decimals), 401, 500.
*I used to do application programming on an AS400 - green screen 5250 terminals. A display screen had its own definition as an input/output file - descriptive fields, inputs, outputs, colour, highlight, etc, and it was declared within the program as a file. The s
Shesh guys... (Score:2)
Did you really have to use that MCAS thingy EVERYWHERE?
Boeing's crewed testing ... (Score:2)
... is crude.
Boeing Software... Oxymoron... (Score:2)
...two words that don't go together.
Where are they getting their software engineers? (Score:2)
I work at a top-tier tech company. I get multiple recruiting emails each year from Google, Amazon, Facebook, and countless startups.
But Boeing has never reached out.