Forgot your password?
typodupeerror
Space EU Technology

Software Error Caused Soyuz/Galileo Failure 157

Posted by samzenpus
from the put-that-anywhere dept.
schwit1 writes An investigation into the recent failed Soyuz launch of the EU's Galileo satellites has found that the Russian Fregat upper stage fired correctly, but its software was programmed for the wrong orbit. From the article: "The failure of the European Union’s Galileo satellites to reach their intended orbital position was likely caused by software errors in the Fregat-MT rocket’s upper-stage, Russian newspaper Izvestia reported Thursday. 'The nonstandard operation of the integrated management system was likely caused by an error in the embedded software. As a result, the upper stage received an incorrect flight assignment, and, operating in full accordance with the embedded software, it has delivered the units to the wrong destination,' an unnamed source from Russian space Agency Roscosmos was quoted as saying by the newspaper."
This discussion has been archived. No new comments can be posted.

Software Error Caused Soyuz/Galileo Failure

Comments Filter:
  • I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it. They obviously haven't even heard of SQA. What gives?

    • by Anonymous Coward

      All their big league programmers are hacking banking and credit card systems, doing black hat stuff. Space program stuff is now for the people who couldn't get the good jobs.

    • Do not underestimate the Russian programmers. If you have not seen their "vaunted chops" you are fortunate.
    • Re: (Score:1, Insightful)

      by Anonymous Coward

      I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it.

      I've heard American programmers are brilliant but then Mars probe crashed because it used wrong units (why didn't it warn that parameter was too low?) ... or the "cloud" services crashed due to (leap year, HD error, "unspecified error", etc.. etc..)

      I've heard European programmers are brilliant, but then Ariane explodified itself due to an overflow

      I've heard Japanese programmers are brilliant, but then the Honda thing happened, causing cars to go out of control.

      They obviously haven't even heard of SQA. What gives?

      Easy to blame things in hind sight and be all g

      • by Smerta (1855348)

        I agree with the sentiment about programming skill, but I think Toyota, not Honda, had the more significant unintended acceleration issues (according to CBS News [cbsnews.com] and NHTSA [rankingsandreviews.com], as many as 89 deaths).

        • My Toyota had this issue. The gas pedal would just stay at full throttle. I lurched my way to the dealer to get it fixed and the mechanics had a good laugh at my expense. The floor mat was ripped and catching up in the pedal. 1 new floor mat later and I was on my way. This was in about 1999.
    • by Brannon (221550) on Thursday August 28, 2014 @11:23PM (#47781457)

      There's almost no overlap between the skills & techniques necessary to write & verify critical software (e.g. when lives or huge amounts of money are on the line) vs. what is considered to be "programming". Modern software engineering's approach to reliable system design is about where hardware engineering was fifty years ago, and about where civil engineering was 100 years ago.

      SQA is a joke. Reliable systems are made using way more robust techniques, including: (a) a severely restricted state space, (b) redundancy, (c) formal proofs, (d) fully (and formally) specified interfaces, (e) random simulation, (f) several different types of coverage, (g) physics-based analysis, etc.

      The failure of the software community to understand this distinction is why I'm scared to death about the coming world of driver-less cars and robots performing surgery. How many people are going to be killed by C++ in the next decade?

      • by ShanghaiBill (739463) on Thursday August 28, 2014 @11:38PM (#47781515)

        I'm scared to death about the coming world of driver-less cars and robots performing surgery.

        Your fears are not rational. Self driving cars and robotic surgeons are tested for thousands of hours, under live conditions. SDCs are not perfect, but they already have a far better safety record than the average human driver. I had LASIK eye surgery done by a robot. I trusted it far more than I would a human surgeon. Getting rocket software right is difficult precisely because there is no way to do a live test. It has to work perfectly on the very first attempt. Very few other applications have such a severe constraint.

        How many people are going to be killed by C++ in the next decade?

        A lot fewer than would have died without it.

        • by Brannon (221550) on Friday August 29, 2014 @12:14AM (#47781675)

          This is probably something that is well understood by the engineers who are building robot surgeons (and maybe even by those building driverless cars), but it certainly isn't well understood by the overwhelming majority of software engineers and it's just a matter of time until the unwashed hordes of C++ monkeys are unleashed unto critical systems.

          Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. Neither are buildings or pacemakers or computer chips.

          There are some scary problems with how [many if not most] software engineers see the world which don't bode well for a world where software can kill:
          (a) by and large they've had essentially no exposure to any method of verification other than "trial & error"
          (b) they have insufficient reverence for cause and effect because most of their bugs have really low cost (as in, nobody dies)--therefore they aren't mentally trained to make disciplined decisions.
          (c) arrogance: unlike every other kind of engineer, software engineers rarely encounter the boundaries of their knowledge. A civil engineer knows when to call a materials engineer, a mechanical engineer knows when to talk to an industrial or chemical engineer, but a software engineer spends their entire lives inside a carefully constructed virtual world where they can't really do that much damage.

          • by ShanghaiBill (739463) on Friday August 29, 2014 @01:14AM (#47781895)

            it's just a matter of time until the unwashed hordes of C++ monkeys are unleashed unto critical systems.

            No way. The corporate lawyers will never let that happen. Neither will the regulators. It is very hard to certify a SDC for public roads. Reams of test data are required. It is even more difficult to get a medical device approved by the FDA. Therac-25 [wikipedia.org] happened almost 30 years ago, a lot of lessons were learned, and it hasn't happened again.

            Bridges aren't designed and tested by "trial & error" ... Neither are buildings or pacemakers or computer chips.

            I have never designed a bridge or pacemaker, but I have designed computer chips. I sit at a workstation, and I type Verilog code into Emacs. It is the same process as writing software, which is mostly trial and error. I write unit tests, do regression testing, etc. I watch it fail, I fix the bugs, and I iterate. Once I get all the bugs fixed, I load it into an FPGA, and watch it fail with some signal skew that I didn't think of. So I write more tests, and repeat. When it runs flawlessly on the FPGA, I ask a co-worker to test it some more, and review my code. Eventually we go to silicon, where a bug costs a million bucks. Usually everything is fine, but that isn't because it is "different" than doing software. It is basically the same process. It is more reliable because most ICs are far less complicated than even a typical iPhone app. They tend to have lots of the same cells repeat over and over. So an IC with a million gates isn't like a million lines of code. It is more like a few dozen 50 line subroutines, that are called a million times.

            • by jythie (914043)

              It is even more difficult to get a medical device approved by the FDA. Therac-25 [wikipedia.org] happened almost 30 years ago, a lot of lessons were learned, and it hasn't happened again.

              This alone will keep the 'unwashed masses' out of such fields. Working with medical systems (or to a lesser degree, any embedded system that does not pretend to be a mini-desktop for consumers) does not have the lax attitude and instant gratification that most programmers coming out of school have grown to expect out of projects. The work is not as sexy, the tools are less likely to be bleeding edge, and for people hoping to go into something 'cool' such work is a bit of a resume stain. Which I suspect s

              • I got "volunteered" to make a medical device because no one else at work was good with the hardware interfaces and it was a HUGE PITA. FDA doesn't like this, they don't like that, test this 500 more times, etc. etc.....
            • by jhol13 (1087781)

              When has a layer done anything more than prevented use of public domain code "because it might be a problem"? I have never heard of a "regulator". I should have, if there is one.

              As is bloody obvious, space system software is programmed by the same cowboy coders as web pages. I do not believe medical equipment are done a bit differently.

              Your "once I get all the bugs fixed" was funny, though, thanks!

              Someone in this thread mentioned formal proofs - they are going to increase by two orders of magnitude next few

          • by readin (838620)
            Most programmers and software engineers have the limitations you mention because consumers don't want to pay for the high quality software we want to build for them. People who go into the field tend to have some OCD-like traits, and making 'perfect' software is what they want to do. But we're not given time so we learn to take short-cuts.

            When software is used in places where it has to work the first time, we'll be more than happy to adapt to the new set of circumstances. There will likely be a few gl
            • Most programmers and software engineers have the limitations you mention because consumers don't want to pay for the high quality software we want to build for them.

              I would lend more credence to this statement if most software engineers actually had any actual experience designing and implementing high reliability software. Most unfortunately have very little clue what that actually means or how to do it for real. They like the idea (it's a good idea!) but have zero experience or training in the implementation techniques required. You are correct that there is an economic component to the problem but that doesn't appear to be the core problem. Even when we take mon

          • Re: (Score:2, Insightful)

            by Anonymous Coward

            The requirements for functional safety in programming industrial safety critical systems are well known, and are very different from the requirements for programming. Boiler flame safety systems are commonly microprocessor based now, and rarely if ever fail. Here are some links explaining some of the requirements.

            PLC® vs. Safety PLC – Fundamental and Significant Differences [rockwellautomation.com]
            FM Global Class 7605 Approval Standard for Programmable Logic Controller Based burner Management Systems [fmglobal.com]
            IEC 61508 Functiona [mtl-inst.com]

            • by tibit (1762298)

              Yet SIL 4 is still hardwired :)

            • Exactly, and for every other industry where you have critical systems that are microprocessor based you have equivalent frameworks. There are 1000's of CPUs in your average rocket or aircraft. Their software is all designed to EXTREMELY tight specifications and under very specific processes and rules. The teams of people who do this work are experienced, stable, very skilled, and have implemented very exacting processes mandated by the FAA or military. The sorts of failures the Russians are having just don

          • by jythie (914043)
            Part of the problem is that, in a way, there is no such thing as a 'software engineer'. There are degrees that call themselves that, and often they are in the right direction, but I would still not put them in the same category as engineering degrees that lead to professional certification.

            Or I guess more accurately, because there is no certification process, because the field is so informal, it is hard to tell from one's title how close they are to CS vs engineering in how they were trained and continue
          • unlike every other kind of engineer, software engineers rarely encounter the boundaries of their knowledge

            I'm not so sure about this. I agree about the arrogance but I think a more accurate statement might be that software engineers too often fail to recognize when they encounter the boundaries of their knowledge. I think they bump into those limits all the time and go merrily on their way past them. We've all seen software that was clearly developed by someone who clearly never actually had to use it to do the job it was designed for. I was staring at a piece of accounting software today that clearly was

          • Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. Neither are buildings or pacemakers or computer chips.

            Should we also assume that rockets are programmed with the same careful methods you (conveniently) omitted?

            Rockets are known to never fail, after all.

          • by cwsumner (1303261)

            ... Bridges aren't designed and tested by "trial & error"--if they were then half of them would fall down within a few weeks. ...

            Bridges were indeed built this way in the 1800's, during the rapid expansion of the US. And many of then did indeed fall down!

        • by antdude (79039)

          I do SQA testings, and I don't trust computers these days.

        • by Anonymous Coward

          You've missed the whole point. The parent is not saying software is 'dangerous', he is saying development techniques we use for software today are not a engineering with required and well understood formal proofs. We don't call out software people 'engineers', they aren't. They are 'developers'. Engineers have a design flow that allows for formal verification, writing software is like writing a novel.

          • by jythie (914043)
            That sounds like a self fulfilling prophecy if I ever heard one. Plenty of universities put out actual software engineers. They go through the same coursework as electrical engineers and are taught formal methods of validation and such. However if your company culture sees software people as 'just developers' and not 'engineers', I would wager that the people who do have the formal training are going to pass your place by. Nobody likes working with people who are not going to consider them 'real', esp w
            • You'll find that most 'software engineering' degrees are from two year diploma mills.

              The variation on EE is typically ComputerE not software E.

        • by Type44Q (1233630) on Friday August 29, 2014 @01:18AM (#47781915)

          Your fears are not rational.

          Just because he's paranoid doesn't mean C++ isn't out to get him...

        • > Getting rocket software right is difficult precisely because there is no way to do a live test.

          As someone who tested software for the Space Station, there is, but it's very expensive, and seldom done. In addition to the other SQA methods mentioned, we had a simulation & test lab next to the clean room where the actual modules were assembled. We simulated all the inputs to the flight computers as if the rest of the Station was there, and flying, including testing all the possible fault conditions

          • by jythie (914043)
            I am always surprised when I go to interview at a company and their test team is actually smaller then their development one.
        • > How many people are going to be killed by C++ in the next decade?

          Depends on how many missile launch systems, drones and other high-tech weaponry was implemented in C++.
        • Your fears are not rational. Self driving cars and robotic surgeons are tested for thousands of hours, under live conditions.

          Among the other parts of my job I run a Quality Assurance department for my company and I've worked in QA for several years. It doesn't matter how much you test something if the process for designing and building the product was inadequate. QA testing is like the goalie on a hockey team - necessary but even the best goalie is going to fail if the team in front of him can't play defense. Good quality comes from good designs which are rationally and systematically well executed. Testing is a part of the e

        • by tibit (1762298)

          "Getting rocket software right is difficult precisely because there is no way to do a live test." There is. You do hardware-in-the-loop tests where the inertial and other inputs come from simulators. I have seen testing of a jet engine controller done without an actual jet engine attached to it. There was a beefy server that was simulating the physics of a jet engine, though, and providing sensor readings.

        • Classical fallacy. The safety records for human drivers include each and every moron who drives piss drunk or under other drugs or simply cannot drive. Unless you're one of those, they will give you almost no useful information for deciding whether you should consider SDCs safe in comparison to your driving skills. And by adding personal anecdotes you confirm the OPs point even more.

        • but they already have a far better safety record than the average human driver.

          I want you to realize that the only source we have for this is Google. It's not from a scientific journal, or an independent research team, or an auditor, it is from the same people who want you to eventually buy their product.

          Furthermore, I want you to realize that the Google team is very careful in what information they reveal. All the information they present is shaped in a way that makes them look good, and to increase demand for the car. Now, maybe they've built the perfect driverless car, and it som

      • by gl4ss (559668) on Friday August 29, 2014 @12:00AM (#47781613) Homepage Journal

        it seems to me that in this case the programmers job was done 100% perfect.

        but the program was given wrong place to take the satellites to.

        • by Yoda222 (943886)
          From what I understand (but this article is reporting that some press agency report that someone unamed told them something... the result must be very accurate) it's more likely that a software A gives a bad result to the Fregat control software B. I'm not able to determine from the article is the software A is bad or if there was a bad input to software A which result in this. I understand from the article that software A is also embedded, but see my remark in the parenthesis before about how accurate this
          • by jythie (914043)
            That is something I am not clear on from the piece either. However, the fact it got to a reasonable orbit even with an error (regardless of if it was an initial input or corrupted by an upstream process) is pretty significant. It ended up in the wrong orbit instead of something catastrophic.
            • Well, it sounded pretty clear that the actual flight CONTROL software worked fine and executed a program perfectly, it just wasn't the program that was intended due to SOME sort of issue with another piece of 'management' software. It sounded like that was also in the spacecraft, but its just as likely it was something running at ground control (makes more sense to me, generally you only run the least amount of software onboard that you can, why waste money on CPU and etc that could be cheaper ground stuff?

      • by radtea (464814)

        How many people are going to be killed by C++ in the next decade?

        None. However, a few will be killed by C++ programmers. I say this as someone who has written code to guide surgeons, but my mantra was: "The surgeon is in control", and I have in fact seen surgeons over-ride the guidance information the software gives them.

        Driverless cars are actually less likely than humans to screw up, I think, but it'll take another decade to prove that. Software engineering is still a nascent field, but in another generation or three it will be at a point where we can be somewhat confi

      • by Anonymous Coward

        Huh, last thing the news said it was programmed for the wrong orbit. Sounds like the software was fine... It did what it was told.

        Programmed for the wrong orbit does not mean bad software programming... So it sounds like the errorthe tfa is taking about was in configuration management, which is definitely lacking nowadays in the world of fast paced social/cloud projects.

      • by R3d M3rcury (871886) on Friday August 29, 2014 @12:40AM (#47781767) Journal

        How many people are going to be killed by C++ in the next decade?

        4.

        I always find the "how many people will be killed" / "how many people have to die before" statements can be answered with this number.

      • by antdude (79039)

        Yeah, I also don't trust computers but humans make (mistake/error)s too. We need better development and testings. :(

      • What's a "Programmer"? Also precisely who should we get to write critical software? A maths teacher? The after hours cleaner? Maybe some random MBA from middle management? Programmers most definitely SHOULD be the ones writing critical software. It's when it is written by non-programmers or hobby programmers with full time other careers (physicists, engineers, etc) that you end up with some of the most basic mistakes and unexpected behaviour.

        Your big mistake is to assume that all programmers are the same, a

      • by Anonymous Coward

        "... where hardware engineering was fifty years ago, and about where civil engineering was 100 years ago ..."

        If your goal was to insult the engineers, you have succeeded.

        For comparing software development with civil engineering, you probably have to select a pre-Roman building.

        And hardware design was always miles ahead from engineering point of view.
        Modern out-of-order processors were created because software was not able to give the same results. So the engineers finally got frustrated enough and solved th

      • Ah, I just love these sorts of pronouncements.

        When was the last time you heard of a 747 crashing because of a software glitch? My first job was to verify the design and implementation of a major part of the flight software for that aircraft, so I'm kind of an expert on this subject. You have no idea how multi-faceted and sophisticated the verification and SQA processes are on these projects. First of all formal logical methods are used to design and validate all the control algorithms. Then the actual syste

      • And yet another argument turns into a C++ hatefest.

    • I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it. They obviously haven't even heard of SQA. What gives?

      You're assuming this wasn't intentional.

    • by AftanGustur (7715)

      I've been hearing all this about the much vaunted chops of these Russian coders, but frankly I don't ever see it.

      There is also the possibility that the project was sabotaged by an external actor.

      Maybe it is a coincidence but the one who profits the most from this failure is the same as has been working hard during the last 10 years to get rid of the Galileo program and is also the same nation as is known for being the most technically capable in electronic warfare/hacking.

    • You can classify programmers into many categories. Two of them are those that write really complex code that is hard to read and not easy to maintain, and they say they are brilliant because no one else can read it / figure it out (easily that is). They may also make it explicitly convoluted and take extra steps to make it more complicated that it needs to be, unbeknownst to them. Then there are the experienced programmers that write easy to read, modular and maintainable code because they don't want to hav
    • cracking US banks.

      so their day jobs, launching rockets and calculating how far Russia could go in subjugating Ukraine, suffered.

      we all know how that works...

  • by msauve (701917) on Thursday August 28, 2014 @10:25PM (#47781243)
    A software error in Russian GLONASS receivers has resulted in thousands of Russian troops innocently crossing the border into Ukraine.
    • by rubycodez (864176) on Thursday August 28, 2014 @10:47PM (#47781323)

      two months ago software error in a 9M317 missile controlled by a BUK missile system rendered it unable to avoid being struck in midair by the careless pilot of Malaysian Airlines Fligh 17MH. Sadly, the missile was a total loss.

  • by x0ra (1249540) on Thursday August 28, 2014 @10:27PM (#47781251)
    From the linked article (emphasis mine): "Galileo Satellites Incident Likely Result of Software Errors", there is still an uncertainty. Though, I guess I should not be surprised, this is /. afterall....
    • by Megane (129182)
      I wouldn't call it a software error if it was "programmed for the wrong orbit". That sounds more like a human error to me. But to be fair, TFA doesn't state that in such specific words.
  • Pfffft (Score:5, Funny)

    by Tablizer (95088) on Thursday August 28, 2014 @10:34PM (#47781277) Homepage Journal

    It's not like it's rocket science to get it right

  • by theycallmeB (606963) on Thursday August 28, 2014 @10:34PM (#47781279)
    the strategic value of satellite navigation and general asshole-erly at the top of the Russian government, I am guessing that Europe's very expensive satellites ended up exactly where Russia wants them.
    • bingo.
    • by Guppy06 (410832) on Thursday August 28, 2014 @11:10PM (#47781403)
      The Russian GLONASS has its own problems [bbc.com], and the whole point of Galileo is a GNSS that is independent from the US. Do you think the Russians like falling back on US technology? Or do you think they're planning to rely on Beidou?
      • I think they would be very happy if the rest of Europe were utilizing GLONASS, a system they can shut down or manipulate if they need to. There's a reason for the four different sat-nav systems currently under operation or construction: No country wants to be dependent on a system operated by someone else. It follows that they would like other countries to be dependent upon theirs.

        • by Guppy06 (410832)

          I think they would be very happy if the rest of Europe were utilizing GLONASS, a system they can shut down or manipulate if they need to.

          But they themselves can't get it to work, as was highlighted in the link I posted. So why would a Europe unable to deploy Galileo use GLONASS instead of GPS?

          And it certainly isn't like Europe doesn't have its own space launch capabilities.

          Russia has absolutely nothing to gain and much to lose by trying to fuck with this launch.

  • Not A SW error! (Score:5, Insightful)

    by Anonymous Coward on Thursday August 28, 2014 @11:08PM (#47781385)

    This is not a SW error! The software put them right where they were told to. The orbital parameters were wrong! This is a data error not a SW error!

    • by mirix (1649853)

      Bingo. If someone sets a GPS to go to the wrong location, you don't say the GPS had a embedded software problem.

      More like a failure to double check settings or something.

      • by qpqp (1969898)

        More like a failure to double check settings or something.

        - "Are you really sure you want to trash those two satellites?"
        <click>
        - "Did you get your boss's approval?"
        <click>

        • by Kittenman (971447)

          More like a failure to double check settings or something.

          - "Are you really sure you want to trash those two satellites?" <click> - "Did you get your boss's approval?" <click>

          Or... the Russian version of Clippy,..

          "Hi - it looks like you're trying to trash two satellites. Do you want a hand with that?"

          <click>

        • by tomhath (637240)
          The satellites weren't trashed. They are in a perfectly good orbit, just the wrong orbit for their intended use.
    • by Yoda222 (943886)
      That's not what I understand from the article (my understanding of the article don't exclude your interpretation, but I see it differently) You are right, the control software of the Fregat did exactly what it was told to do (assuming the article is correct) but the input was wrong. But was the input directly done by an human or is it the result of another software (more likely) which had an error? If this is the case, then it's a software error. (this could also be an human input error before this software
    • by jythie (914043)
      I am actually rather impressed that the software managed a stable orbit even with user error. Yet this has turned into who knows how many posts of people complaining about how bad 'programmers' are.

      In a way this reflects how the blame game in development often seems to work. Programmers are faced with shifting requirements, tight deadlines, undersized testing teams, pressure to work hours that result in fatigue and higher error rates, decisions being made by marketers and MBAs who do not understand the c
      • by Rich0 (548339)

        Programmers are faced with shifting requirements, tight deadlines, undersized testing teams, pressure to work hours that result in fatigue and higher error rates, decisions being made by marketers and MBAs who do not understand the consequences of various changes, but blame tends to fall on the programmers for writing buggy software.

        Meh, it seems that more and more the blame is shifting to whoever wrote the requirements or the project manager. But, without fixing all that other stuff, there isn't much they can do either.

        The whole reason Agile was created was to try to deal with some of these pressures, which many consider unavoidable. The problem is that many companies just don't grok it, and insist on a fixed scope in a fixed time at a fixed cost. This approach fits in better with annual planning cycles. If you have a million doll

  • by Tablizer (95088) on Friday August 29, 2014 @02:54AM (#47782223) Homepage Journal

    They had trouble putin it in the right orbit

  • The software worked perfectly. This is a case of misprogrammed destination ("What do you mean this is Auckland? I wanted OAKLAND!")

  • Probably.

  • What's the old saying? There is no izvestia in Pravda, and no pravda in Izvestia.
  • Revenge is a dish best served in the wrong orbit?

    Funny, isn't it, in the midst of all these sanctions and general brou-ha-ha over the Ukraine, with Russia taking all kinds of tit-for-tat punitive measure in response by EU attempts use economic fines in order to restrain their bad behavior, that, âoeThe nonstandard operation of the integrated management system was likely caused by an error in the embedded software," which manages to cost the EU the full use of a multi-million dollar satellite whose p

C'est magnifique, mais ce n'est pas l'Informatique. -- Bosquet [on seeing the IBM 4341]

Working...