Software Error Caused Soyuz/Galileo Failure 157
schwit1 writes An investigation into the recent failed Soyuz launch of the EU's Galileo satellites has found that the Russian Fregat upper stage fired correctly, but its software was programmed for the wrong orbit. From the article: "The failure of the European Union’s Galileo satellites to reach their intended orbital position was likely caused by software errors in the Fregat-MT rocket’s upper-stage, Russian newspaper Izvestia reported Thursday. 'The nonstandard operation of the integrated management system was likely caused by an error in the embedded software. As a result, the upper stage received an incorrect flight assignment, and, operating in full accordance with the embedded software, it has delivered the units to the wrong destination,' an unnamed source from Russian space Agency Roscosmos was quoted as saying by the newspaper."
Russian Programmer's are Brilliant! (Score:2)
I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it. They obviously haven't even heard of SQA. What gives?
Re: (Score:1)
All their big league programmers are hacking banking and credit card systems, doing black hat stuff. Space program stuff is now for the people who couldn't get the good jobs.
Re: (Score:2)
Re: (Score:3)
Ukraine is Russia now
Re: (Score:1)
The Beatles reported that the Ukraine girls really knocked them out. Said they left the West behind.
Re: (Score:3)
Yeah, but Moscow girls made them sing and shout.
Re: Russian Programmer's are Brilliant! (Score:1)
Re: (Score:2)
Re: (Score:2)
Another satellite state that has fallen into low orbit.
Re: (Score:1, Insightful)
I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it.
I've heard American programmers are brilliant but then Mars probe crashed because it used wrong units (why didn't it warn that parameter was too low?) ... or the "cloud" services crashed due to (leap year, HD error, "unspecified error", etc.. etc..)
I've heard European programmers are brilliant, but then Ariane explodified itself due to an overflow
I've heard Japanese programmers are brilliant, but then the Honda thing happened, causing cars to go out of control.
They obviously haven't even heard of SQA. What gives?
Easy to blame things in hind sight and be all g
Re: (Score:2)
I agree with the sentiment about programming skill, but I think Toyota, not Honda, had the more significant unintended acceleration issues (according to CBS News [cbsnews.com] and NHTSA [rankingsandreviews.com], as many as 89 deaths).
Re: (Score:2)
Re: (Score:2)
Some cases were due to people pressing the wrong pedals. But some were also due to a the software screwing up - http://www.sddt.com/Commentary... [sddt.com]
Re: (Score:2)
The software was certainly not the best, but considering that most "unintended acceleration" events happened after a couple of nasty accidents (caused by faulty/incorrect floor mats, not a design flaw in the throttle system), I don't buy it, other than for maybe a freak instance.
The vast majority was made up of people who either maliciously covered up their mistakes by blaming the car or should not have been driving at all and genuinely believed they had done nothing wrong.
Re: (Score:2)
Sure, media buzz meant people blamed their errors on that. However, that doesn't mean there wasn't also a problem. That Toyota lost that particular court case seems a reasonable indication there was in fact a problem.
"Programmers" shouldn't write critical software (Score:5, Insightful)
There's almost no overlap between the skills & techniques necessary to write & verify critical software (e.g. when lives or huge amounts of money are on the line) vs. what is considered to be "programming". Modern software engineering's approach to reliable system design is about where hardware engineering was fifty years ago, and about where civil engineering was 100 years ago.
SQA is a joke. Reliable systems are made using way more robust techniques, including: (a) a severely restricted state space, (b) redundancy, (c) formal proofs, (d) fully (and formally) specified interfaces, (e) random simulation, (f) several different types of coverage, (g) physics-based analysis, etc.
The failure of the software community to understand this distinction is why I'm scared to death about the coming world of driver-less cars and robots performing surgery. How many people are going to be killed by C++ in the next decade?
Re:"Programmers" shouldn't write critical software (Score:5, Insightful)
I'm scared to death about the coming world of driver-less cars and robots performing surgery.
Your fears are not rational. Self driving cars and robotic surgeons are tested for thousands of hours, under live conditions. SDCs are not perfect, but they already have a far better safety record than the average human driver. I had LASIK eye surgery done by a robot. I trusted it far more than I would a human surgeon. Getting rocket software right is difficult precisely because there is no way to do a live test. It has to work perfectly on the very first attempt. Very few other applications have such a severe constraint.
How many people are going to be killed by C++ in the next decade?
A lot fewer than would have died without it.
Testing is not verification. (Score:4, Insightful)
This is probably something that is well understood by the engineers who are building robot surgeons (and maybe even by those building driverless cars), but it certainly isn't well understood by the overwhelming majority of software engineers and it's just a matter of time until the unwashed hordes of C++ monkeys are unleashed unto critical systems.
Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. Neither are buildings or pacemakers or computer chips.
There are some scary problems with how [many if not most] software engineers see the world which don't bode well for a world where software can kill:
(a) by and large they've had essentially no exposure to any method of verification other than "trial & error"
(b) they have insufficient reverence for cause and effect because most of their bugs have really low cost (as in, nobody dies)--therefore they aren't mentally trained to make disciplined decisions.
(c) arrogance: unlike every other kind of engineer, software engineers rarely encounter the boundaries of their knowledge. A civil engineer knows when to call a materials engineer, a mechanical engineer knows when to talk to an industrial or chemical engineer, but a software engineer spends their entire lives inside a carefully constructed virtual world where they can't really do that much damage.
Re:Testing is not verification. (Score:5, Informative)
it's just a matter of time until the unwashed hordes of C++ monkeys are unleashed unto critical systems.
No way. The corporate lawyers will never let that happen. Neither will the regulators. It is very hard to certify a SDC for public roads. Reams of test data are required. It is even more difficult to get a medical device approved by the FDA. Therac-25 [wikipedia.org] happened almost 30 years ago, a lot of lessons were learned, and it hasn't happened again.
Bridges aren't designed and tested by "trial & error" ... Neither are buildings or pacemakers or computer chips.
I have never designed a bridge or pacemaker, but I have designed computer chips. I sit at a workstation, and I type Verilog code into Emacs. It is the same process as writing software, which is mostly trial and error. I write unit tests, do regression testing, etc. I watch it fail, I fix the bugs, and I iterate. Once I get all the bugs fixed, I load it into an FPGA, and watch it fail with some signal skew that I didn't think of. So I write more tests, and repeat. When it runs flawlessly on the FPGA, I ask a co-worker to test it some more, and review my code. Eventually we go to silicon, where a bug costs a million bucks. Usually everything is fine, but that isn't because it is "different" than doing software. It is basically the same process. It is more reliable because most ICs are far less complicated than even a typical iPhone app. They tend to have lots of the same cells repeat over and over. So an IC with a million gates isn't like a million lines of code. It is more like a few dozen 50 line subroutines, that are called a million times.
Re: (Score:2)
It is even more difficult to get a medical device approved by the FDA. Therac-25 [wikipedia.org] happened almost 30 years ago, a lot of lessons were learned, and it hasn't happened again.
This alone will keep the 'unwashed masses' out of such fields. Working with medical systems (or to a lesser degree, any embedded system that does not pretend to be a mini-desktop for consumers) does not have the lax attitude and instant gratification that most programmers coming out of school have grown to expect out of projects. The work is not as sexy, the tools are less likely to be bleeding edge, and for people hoping to go into something 'cool' such work is a bit of a resume stain. Which I suspect s
Re: (Score:2)
Re: (Score:2)
When has a layer done anything more than prevented use of public domain code "because it might be a problem"? I have never heard of a "regulator". I should have, if there is one.
As is bloody obvious, space system software is programmed by the same cowboy coders as web pages. I do not believe medical equipment are done a bit differently.
Your "once I get all the bugs fixed" was funny, though, thanks!
Someone in this thread mentioned formal proofs - they are going to increase by two orders of magnitude next few
Re: (Score:2)
When software is used in places where it has to work the first time, we'll be more than happy to adapt to the new set of circumstances. There will likely be a few gl
Most developers only know trial and error (Score:2)
Most programmers and software engineers have the limitations you mention because consumers don't want to pay for the high quality software we want to build for them.
I would lend more credence to this statement if most software engineers actually had any actual experience designing and implementing high reliability software. Most unfortunately have very little clue what that actually means or how to do it for real. They like the idea (it's a good idea!) but have zero experience or training in the implementation techniques required. You are correct that there is an economic component to the problem but that doesn't appear to be the core problem. Even when we take mon
Re: (Score:2, Insightful)
The requirements for functional safety in programming industrial safety critical systems are well known, and are very different from the requirements for programming. Boiler flame safety systems are commonly microprocessor based now, and rarely if ever fail. Here are some links explaining some of the requirements.
PLC® vs. Safety PLC – Fundamental and Significant Differences [rockwellautomation.com]
FM Global Class 7605 Approval Standard for Programmable Logic Controller Based burner Management Systems [fmglobal.com]
IEC 61508 Functiona [mtl-inst.com]
Re: (Score:2)
Yet SIL 4 is still hardwired :)
Re: (Score:2)
Exactly, and for every other industry where you have critical systems that are microprocessor based you have equivalent frameworks. There are 1000's of CPUs in your average rocket or aircraft. Their software is all designed to EXTREMELY tight specifications and under very specific processes and rules. The teams of people who do this work are experienced, stable, very skilled, and have implemented very exacting processes mandated by the FAA or military. The sorts of failures the Russians are having just don
Re: (Score:2)
Or I guess more accurately, because there is no certification process, because the field is so informal, it is hard to tell from one's title how close they are to CS vs engineering in how they were trained and continue
Failing to recognize limits (Score:2)
unlike every other kind of engineer, software engineers rarely encounter the boundaries of their knowledge
I'm not so sure about this. I agree about the arrogance but I think a more accurate statement might be that software engineers too often fail to recognize when they encounter the boundaries of their knowledge. I think they bump into those limits all the time and go merrily on their way past them. We've all seen software that was clearly developed by someone who clearly never actually had to use it to do the job it was designed for. I was staring at a piece of accounting software today that clearly was
Re: (Score:2)
Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. Neither are buildings or pacemakers or computer chips.
Should we also assume that rockets are programmed with the same careful methods you (conveniently) omitted?
Rockets are known to never fail, after all.
Re: (Score:1)
... Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. ...
Bridges were indeed built this way in the 1800's, during the rapid expansion of the US. And many of then did indeed fall down!
Re: (Score:2)
I do SQA testings, and I don't trust computers these days.
Re: (Score:1)
You've missed the whole point. The parent is not saying software is 'dangerous', he is saying development techniques we use for software today are not a engineering with required and well understood formal proofs. We don't call out software people 'engineers', they aren't. They are 'developers'. Engineers have a design flow that allows for formal verification, writing software is like writing a novel.
Re: (Score:2)
Re: (Score:2)
You'll find that most 'software engineering' degrees are from two year diploma mills.
The variation on EE is typically ComputerE not software E.
Re:"Programmers" shouldn't write critical software (Score:5, Funny)
Your fears are not rational.
Just because he's paranoid doesn't mean C++ isn't out to get him...
Re: (Score:2)
> Getting rocket software right is difficult precisely because there is no way to do a live test.
As someone who tested software for the Space Station, there is, but it's very expensive, and seldom done. In addition to the other SQA methods mentioned, we had a simulation & test lab next to the clean room where the actual modules were assembled. We simulated all the inputs to the flight computers as if the rest of the Station was there, and flying, including testing all the possible fault conditions
Re: (Score:3)
Re: (Score:2)
Depends on how many missile launch systems, drones and other high-tech weaponry was implemented in C++.
Hours of testing doesn't equal automatic quality (Score:2)
Your fears are not rational. Self driving cars and robotic surgeons are tested for thousands of hours, under live conditions.
Among the other parts of my job I run a Quality Assurance department for my company and I've worked in QA for several years. It doesn't matter how much you test something if the process for designing and building the product was inadequate. QA testing is like the goalie on a hockey team - necessary but even the best goalie is going to fail if the team in front of him can't play defense. Good quality comes from good designs which are rationally and systematically well executed. Testing is a part of the e
Re: (Score:2)
"Getting rocket software right is difficult precisely because there is no way to do a live test." There is. You do hardware-in-the-loop tests where the inertial and other inputs come from simulators. I have seen testing of a jet engine controller done without an actual jet engine attached to it. There was a beefy server that was simulating the physics of a jet engine, though, and providing sensor readings.
Re: (Score:2)
Classical fallacy. The safety records for human drivers include each and every moron who drives piss drunk or under other drugs or simply cannot drive. Unless you're one of those, they will give you almost no useful information for deciding whether you should consider SDCs safe in comparison to your driving skills. And by adding personal anecdotes you confirm the OPs point even more.
Re: (Score:2)
but they already have a far better safety record than the average human driver.
I want you to realize that the only source we have for this is Google. It's not from a scientific journal, or an independent research team, or an auditor, it is from the same people who want you to eventually buy their product.
Furthermore, I want you to realize that the Google team is very careful in what information they reveal. All the information they present is shaped in a way that makes them look good, and to increase demand for the car. Now, maybe they've built the perfect driverless car, and it som
Re:"Programmers" shouldn't write critical software (Score:5, Insightful)
it seems to me that in this case the programmers job was done 100% perfect.
but the program was given wrong place to take the satellites to.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Well, it sounded pretty clear that the actual flight CONTROL software worked fine and executed a program perfectly, it just wasn't the program that was intended due to SOME sort of issue with another piece of 'management' software. It sounded like that was also in the spacecraft, but its just as likely it was something running at ground control (makes more sense to me, generally you only run the least amount of software onboard that you can, why waste money on CPU and etc that could be cheaper ground stuff?
Re: (Score:2)
How many people are going to be killed by C++ in the next decade?
None. However, a few will be killed by C++ programmers. I say this as someone who has written code to guide surgeons, but my mantra was: "The surgeon is in control", and I have in fact seen surgeons over-ride the guidance information the software gives them.
Driverless cars are actually less likely than humans to screw up, I think, but it'll take another decade to prove that. Software engineering is still a nascent field, but in another generation or three it will be at a point where we can be somewhat confi
Re: (Score:1)
Huh, last thing the news said it was programmed for the wrong orbit. Sounds like the software was fine... It did what it was told.
Programmed for the wrong orbit does not mean bad software programming... So it sounds like the errorthe tfa is taking about was in configuration management, which is definitely lacking nowadays in the world of fast paced social/cloud projects.
How many people... (Score:4, Funny)
How many people are going to be killed by C++ in the next decade?
4.
I always find the "how many people will be killed" / "how many people have to die before" statements can be answered with this number.
Re: (Score:2)
Yeah, I also don't trust computers but humans make (mistake/error)s too. We need better development and testings. :(
What's a "Programmer"? (Score:2)
What's a "Programmer"? Also precisely who should we get to write critical software? A maths teacher? The after hours cleaner? Maybe some random MBA from middle management? Programmers most definitely SHOULD be the ones writing critical software. It's when it is written by non-programmers or hobby programmers with full time other careers (physicists, engineers, etc) that you end up with some of the most basic mistakes and unexpected behaviour.
Your big mistake is to assume that all programmers are the same, a
Re: (Score:1)
"... where hardware engineering was fifty years ago, and about where civil engineering was 100 years ago ..."
If your goal was to insult the engineers, you have succeeded.
For comparing software development with civil engineering, you probably have to select a pre-Roman building.
And hardware design was always miles ahead from engineering point of view.
Modern out-of-order processors were created because software was not able to give the same results. So the engineers finally got frustrated enough and solved th
Re: (Score:3)
Ah, I just love these sorts of pronouncements.
When was the last time you heard of a 747 crashing because of a software glitch? My first job was to verify the design and implementation of a major part of the flight software for that aircraft, so I'm kind of an expert on this subject. You have no idea how multi-faceted and sophisticated the verification and SQA processes are on these projects. First of all formal logical methods are used to design and validate all the control algorithms. Then the actual syste
Re: (Score:2)
And yet another argument turns into a C++ hatefest.
Re: (Score:2)
I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it. They obviously haven't even heard of SQA. What gives?
You're assuming this wasn't intentional.
Re: (Score:2)
Re: (Score:2)
No, but they might be interested in a few more customers for GLONASS....
Re: (Score:2)
I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it.
There is also the possibility that the project was sabotaged by an external actor.
Maybe it is a coincidence but the one who profits the most from this failure is the same as has been working hard during the last 10 years to get rid of the Galileo program and is also the same nation as is known for being the most technically capable in electronic warfare/hacking.
Re: (Score:2)
they were spending too much time on the night job (Score:2)
cracking US banks.
so their day jobs, launching rockets and calculating how far Russia could go in subjugating Ukraine, suffered.
we all know how that works...
In other news... (Score:5, Funny)
Re:In other news... (Score:5, Funny)
two months ago software error in a 9M317 missile controlled by a BUK missile system rendered it unable to avoid being struck in midair by the careless pilot of Malaysian Airlines Fligh 17MH. Sadly, the missile was a total loss.
Re: (Score:2)
The most plausible explanation I've seen so far is that the separatists were supplied with the missiles but, due to the need for Russia to maintain deniability, not an expert in their use - just a crash course in how to fire the things, without full training in target identification. That would explain how they were able to make such an error as mistaking a giant passenger airliner for a small military aircraft.
It's possible that Ukraine shot it down, they use the same missiles, but their army consists of t
Re: (Score:2)
It's possible that Ukraine shot it down, they use the same missiles, but their army consists of trained professional soldiers who would be less likely to make such an error.
... unless the Ukrainians knew it would end up getting blamed on the Russians/separatists which would (and did) cement European support for sanctions against Russia.
Re: (Score:1)
> It's the fucking Malaysian Airlines fault. Nobody in his right mind fly civilian aircraft into a war zone.
Nope. The airspace was opened above a certain altitude. This is not Malaysian Airlines fault. And Malaysian airline was definetely not the only airline to go trhough that route, many others were.
Separatists fired a russian missle at a civilian aircraft. Because they are morons. That's what happened.
Re: (Score:2)
"Legitimate?" Those "separatists" are in violation of the law of their land, they are armed terrorists. Do you think you would be legitimate if you illegally possessed heavy weapons and declared war on Washington DC, and were shooting down civilian aircraft, blowing up vehicles and killing people? Of course not.
Misleading title... once again (Score:3)
Re: (Score:2)
Pfffft (Score:5, Funny)
It's not like it's rocket science to get it right
Re: (Score:2)
Its not even rocket surgery.
Re: (Score:2)
It's not like it's brain surgery
Given current tensions, ... (Score:5, Interesting)
Re: (Score:2)
Re:Given current tensions, ... (Score:5, Interesting)
Re: (Score:2)
I think they would be very happy if the rest of Europe were utilizing GLONASS, a system they can shut down or manipulate if they need to. There's a reason for the four different sat-nav systems currently under operation or construction: No country wants to be dependent on a system operated by someone else. It follows that they would like other countries to be dependent upon theirs.
Re: (Score:2)
I think they would be very happy if the rest of Europe were utilizing GLONASS, a system they can shut down or manipulate if they need to.
But they themselves can't get it to work, as was highlighted in the link I posted. So why would a Europe unable to deploy Galileo use GLONASS instead of GPS?
And it certainly isn't like Europe doesn't have its own space launch capabilities.
Russia has absolutely nothing to gain and much to lose by trying to fuck with this launch.
Re: (Score:2)
A joint effort between many nations who, in the event of conflict, will act as one allied block. Probably alongside the US.
Not A SW error! (Score:5, Insightful)
This is not a SW error! The software put them right where they were told to. The orbital parameters were wrong! This is a data error not a SW error!
Re: (Score:2)
Bingo. If someone sets a GPS to go to the wrong location, you don't say the GPS had a embedded software problem.
More like a failure to double check settings or something.
Re: (Score:2)
- "Are you really sure you want to trash those two satellites?"
<click>
- "Did you get your boss's approval?"
<click>
Re: (Score:3)
- "Are you really sure you want to trash those two satellites?" <click> - "Did you get your boss's approval?" <click>
Or... the Russian version of Clippy,..
"Hi - it looks like you're trying to trash two satellites. Do you want a hand with that?"
<click>
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
In a way this reflects how the blame game in development often seems to work. Programmers are faced with shifting requirements, tight deadlines, undersized testing teams, pressure to work hours that result in fatigue and higher error rates, decisions being made by marketers and MBAs who do not understand the c
Re: (Score:2)
Programmers are faced with shifting requirements, tight deadlines, undersized testing teams, pressure to work hours that result in fatigue and higher error rates, decisions being made by marketers and MBAs who do not understand the consequences of various changes, but blame tends to fall on the programmers for writing buggy software.
Meh, it seems that more and more the blame is shifting to whoever wrote the requirements or the project manager. But, without fixing all that other stuff, there isn't much they can do either.
The whole reason Agile was created was to try to deal with some of these pressures, which many consider unavoidable. The problem is that many companies just don't grok it, and insist on a fixed scope in a fixed time at a fixed cost. This approach fits in better with annual planning cycles. If you have a million doll
Pun Error on Line 472 (Score:3, Funny)
They had trouble putin it in the right orbit
Re: (Score:3)
Re: (Score:3)
Russians have bear hands? I hope if we ever go to war, it's not hand-to-hand combat!
Re: (Score:2)
Garbage in, garbage out (Score:1)
The software worked perfectly. This is a case of misprogrammed destination ("What do you mean this is Auckland? I wanted OAKLAND!")
Written in Scala? (Score:2)
Probably.
Can we trust the report? (Score:2)
In other words, retritbution... (Score:2)
Funny, isn't it, in the midst of all these sanctions and general brou-ha-ha over the Ukraine, with Russia taking all kinds of tit-for-tat punitive measure in response by EU attempts use economic fines in order to restrain their bad behavior, that, âoeThe nonstandard operation of the integrated management system was likely caused by an error in the embedded software," which manages to cost the EU the full use of a multi-million dollar satellite whose p
Re: (Score:2)