Computer Date Glitch May Limit Next Shuttle Launch

n3hat writes "Reuters reports that the next Space Shuttle mission may have to be deferred if it gets too close to the New Year because the onboard computers do not handle the changing of the date in the same way as the ground computers. From the article: '"The shuttle computers were never envisioned to fly through a year-end changeover," space shuttle program manager Wayne Hale told a briefing. The problem, according to Hale, is that the shuttle's computers do not reset to day one, as ground-based systems that support shuttle navigation do. Instead, after December 31, the 365th day of the year, shuttle computers figure January 1 is just day 366."

wtf?

[PhrostyMcByte (589271)]

Is there a reason these aren't built on standard parts and operating systems? If they ran their shuttles on something like Debian stable it would be a rock solid platform and probably end up saving them lots of money. Or am I missing something here.

Re:wtf?

[jbrader (697703)]

Is there a reason these aren't built on standard parts and operating systems?

It was built by the government.

Re:wtf?

[schnikies79 (788746)]

your idea of rock solid and their idea is little different. they have probably one of the most bug-free pieces of software in existence. it's tailored to do what it needs to do, nothing more, nothing less and it does it perfectly.

Re:wtf?

[by Anonymous Coward]

Bug free except for the rollover to a new year...

Xmas at home

[wwwillem (253720)]

This is not a bug ....

Imagine you are a member of the shuttle design team and you can make a choice (for the next 20 years) to either know for sure that you're with the kids at home on X-mas and New Year .... or you can suggest a software feature that could result in your New Year's Eve being spoiled down the road because you have to be for days in a dumb control room. Hey, what would you do??

And I still remember, when I was a kid, that we had that Apollo flight during X-mas. I think it was the one that would for the first time go behind the moon. Someone in the control room that year made it into an important enough person on the Shuttle program so that this WOULD NEVER HAPPEN AGAIN. :-)

Re:wtf?

[Splab (574204)]

You know, people like you give programmers a bad rep. You just dive in for the fix without knowing the cause - and on top of that add a few bugs that are even harder to iron out if you happen to be the only person knowing that code segment.

You're clearly not a very good SA

[multipart/mixed (163409)]

So, what if, oh, say, the CO2 scrubbers need to work differently depending on how many days the mission has been run. So, they keep track of the first day number, and the current day number. The amount of CO2 scrubbing then is varied based on elapsed days.

^^and here's the key -- it's something you don't know about^^

Now, you make your little 5-second fix, and send seven astronauts into space.

New Year's Eve rolls around, and suddenly the mission started on day 360 and it's now day 1. Holy crap, says the scrubber, we have to scrub as though it's a 359-day mission, instead of a lousy 6.

Scrubbers go into overtime, and break. (Or, scrubber math is done in eight bits, and they think the shuttle's still on the ground and not ready to launch for another ~100 days due to integer roll-over. Or any other set of unforseen possibilities.)

Next, astronauts die of CO2 poisoning because the scrubber subsystem has been compromised.

Great fix, mister five-second-coder.

Re:wtf?

[tverbeek (457094)]

Is there a reason these aren't built on standard parts and operating systems? If they ran their shuttles on something like Debian stable it would be a rock solid platform and probably end up saving them lots of money. Or am I missing something here.

Yeah, you're missing something. Such as the fact that the Shuttle was designed a quarter century ago, when Debian was so far from a stable release that Ian hadn't gotten the hange of long division yet, and still believed that Deb had cooties. "Standard parts" meant 8-bit CPUs with 64KB address spaces, and "standard operating systems" included CP/M, 4BSD, VMS, and an upstart known as PC-DOS. I can't really blame them for building something in-house instead.

Re:wtf?

[by Anonymous Coward]

You neglect the fact that military/gov't programming languages at this time included HAL/S, Jovial, NELIAC, &c. (Yes I know that Jovial is still in use for the Navy's ITS 8/16bit muControllers). I've used XPL (the language that the HAL/S compiler was written in) & HAL/S; It was basically the predecessor to Ada/SPARK's 'provability' & 'stability'.
Here is a decent source of HAL/S examples:
http://www.hq.nasa.gov/office/pao/History/computer s/Appendix-II.html [nasa.gov]

Now, look at the procedure called 'read_accel', about 1/4 down the page.
Midway through, there is a ton of gunk. That's the HAL/S maths for you:
the program is allowed to use three lines to express mathematical code, to 'mimic' math
in code. Now, this is the '70s; it's of little wonder that they weren't worried about the
date switch so much as making sure that:
1) the compiler produced code that could be checked & double checked to be '100%' failure proof and at least be resilient to problems.
2) It had to deal with the beast of being machine independent & easily understandable to the PL/I & FORTRAN programmers of the day
3) It also had to make sure that tasks were scheduled properly & run when specific interrupts happened. This is the Norm for Ada/SPARK now, but HAL/S was pretty much the pioneer here within the Aerospace field.

Actually, the numbers differ

[Overzeetop (214511)]

Actually, the estimated failure rate for the shuttle program was 1 in 35, though the shuttles themselves may have been designed to withstand 100 launch/landing cycles*. This was a bit of an issue when the 25th mission resulted in a failure (since most of the population does not understand statistics).

And, for the record, there have been 117 launches, according to wiki, which I will take as accurate enough for this discussion (far less than 200).

*yes, IWAAE (I was an aerospace engineer) working for NASA, and was involved with shuttle payloads and structural reliability analyses.

Re:wtf?

[Schraegstrichpunkt (931443)]

Is there a reason these aren't built on standard parts and operating systems?

Standard parts don't like being bombarded with radiation. Standard operating systems aren't fault-tolerant.

Re:wtf?

[E-Lad (1262)]

You're indeed missing something here.

While I'm not thoroughly educated on this particular subject, I would say that it's a pretty good chance that the flight computers on the shuttles are based on technology that's at least 15 years old (all shuttles underwent a "glass cockpit" update in the mid-late 90s). You don't see NASA cutting a purchase order to cdwg.com when the newest AMD or Intel offering is announced and stuffing that into the shuttles. This stuff is designed, planned, coded for and integrated over a number of years and is very static. No changes. If there has to be changes, they're done under a quality control methods so strict that, yes, Duke Nukem 3D might see the light of day first.

And that's just the hardware part.

On the software side, I'd say you're probably looking at stuff written in any assortment of "classic" languages such as ADA, COBOL, or worse. Due to the nature of the metric f*k ton of sensors, mechanical servos, data inputs, and other such esoteric (and dated) hardware on the shuttles, the software must control, query, parse and monitor, the software is pretty darn married to the platform it runs on.

So, before blurting "D0odz, just instahl leenux n yr shuttlz (deeban stble rox wif glox!)" Give it some deeper thought. There's likely a darn good reason why things are they way they are (bugs not withstanding) when it comes to large flying contraptions that are designed to safely get 7 people 300 miles up, keep them there for a week (or two) and get them home. Sometimes simple things (to you and I) such as a year roll-over are outside the scope when it comes to designing systems to do what the shuttle does.

Well....

[SuperBanana (662181)]

...I guess it -is- rocket science.

*ducks and runs for cover*

Seriously though- they never "envisioned" a mission occuring over the end-of-year? Let me guess: a defense (space) contractor designed the systems.

Well, of course.

[T-Ranger (10520)]

The shuttle runs on three modified IBM 360 systems. Were pushing 35, almost 40 year old systems here.

Do you know how many eligible 35 year old computer bachelors there are out there? Ill tell you: none. Of course the shuttle computers can't get a date.

I'm enjoying a little schadenfreude...

[SchnauzerGuy (647948)]

As a professional software developer, I have heard on countless occasions about how the Space Shuttle software development process [fastcompany.com] is so incredible, and how all other developers should try to live up their high standards.

Granted, the work they do is very impressive and the process is very exacting. But come on...they haven't been able to fix a simple year rollover event in 30 years?!?

From the Fast Company article:

Consider these stats : the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors.

I would say that requiring a reboot every year on December 31 is a pretty huge error. In this case, it is forcing NASA to launch earlier than they otherwise would wish. And this isn't the first time this type of problem has caused problems. The New Scientist has a similar article [newscientistspace.com] that goes into more detail:

This is not the first time that the shuttle programme has been faced with the year-end rollover problem. On a Hubble servicing mission in 1999, the year of the overblown Y2K computer scare, the shuttle landed on 27 December (see Fuel fault delays space repair). To make sure the shuttle got back on the ground before 31 December, mission managers decided to drop one of the four planned spacewalks.

If it ain't broke

[GreggBz (777373)]

So, they made the software so it does not kill anyone. Who needs fancy features like precise yearly timing?

Seriously, though, it's worked fine. The software has not killed anyone. They can either fix it and modify a very critical system on an enormously complex vehicle, or they can move the launch date around a few days, which they seem to do for every launch anyway. B is probably safer and more predictable.

Date/Time Formats

[Detritus (11846)]

The problem is that NASA, and other space agencies, standardized on a date/time format composed of day-of-year (1..366) and time-of-day (UTC). This goes back to the 1960s. In ASCII, the clock looks like "310 04:35.27.642". This date/time format is embedded in a huge amount of hardware, software and standards documents. It's also used for things like countdown clocks and MET (mission elapsed time) clocks.

The end-of-year rollover depends on the leap year and leap second (if any), and has traditionally been a source of problems.

Hold up, everybody

[Alizarin Erythrosin (457981)]

I work with military navigation software, and that is sorta remotely applicable to this. Here's my thoughts:

You people with your "WTF NASA SUXORS THIS IS EASY FIX!!!11!!1!one!!" need to stop and think for a second. This is a space application that carries HUMAN BEINGS! Think about how hard it will be to get this "easy fix" qualified, proven, documented, etc. Its not an easy task. A formal qualification test on the systems I work on (military land- and air-, but not space-based navigation software) can take months, and require all sorts of tests and documentation. Anything that isn't formally tested (i.e. run in a van, on a plane, etc) must be shown to not fail in any way; all exceptions handled, no bad data can cause an undesireable state, etc. I would hate to see the type of scrutiny that the Shuttle software goes through (although I could probably call somebody in our Space division across the street and find out).

Second, I don't know exact specifics, but based on the information provided, I think this "glitch" will have to do with the data/time difference between ground stations and the Shuttle computers. Things like message time stamping between the Earth and the Shuttle, etc, will be wrong, and things could be garbled or just dropped all together. The navigation systems themselves should not be terribly impacted since the date will just roll to the next day. Inertial instrument samples will continue to flow in and be correctly time stamped, be it the 366th or 400th or 500th day.

No CMMI comments? Are there real developers here?

[tacokill (531275)]

Hundreds of comments and not a single one mentions that NASA is a CMMI Level 5 organization. For those that don't know (and apparently, that's a lot of you), CMMI, aka Capability Maturity Model Integration, is software ENGINEERING methodology for developing processes and technologies around IT systems. It is a very in-depth methodology for developing software and comes about as close to "engineering" as you can get in software development.

Here is a list of participants in this program. [cmu.edu]

And here [cmu.edu] is a general overview of what CMMI is.

And just to put it into perspective, when I was last working with CMMI, there were only 3 companies certfied at level 5. Nasa, Motorola, and another one I can't remember. I am sure that has changed but nonetheless, it's a big deal and shows a serious effort to do things in a controlled, measureable, testable, way.

I only bring this up to counter the ridiculous "solutions" that some have proposed on this site.

"I can fix that in 3 lines of code".

Well, great. That might work at YOUR company. But please don't do that at NASA. Despite what many think here, NASA is a top-notch software development house. And I would expect nothing less given what is at stake.

Re:lame

[morgan_greywolf (835522)]

(or maybe he watched too much Star Trek that he thought he should follow the intergalactical star dates)

Oh, shit! You mean we're not supposed to be following intergalactic star dates?? No wonder those programs I wrote have so many date bugs...

   

Re:lame

[ozmanjusri (601766)]

Well we know that programmers get confused with numbers one time or another since we're used to start things at index 0.

Nah, everyone knows geeks are useless at dates because they never get any. Predictable failure, that one.

Re:How Many Times?

[WindBourne (631190)]

Perhaps the fact that the shuttle was being developed at the same time as ADA might have something to do with it. [umich.edu] Or do you recommend using a not even fully designed, coded, and tested language for controlling the most complex piece of equipment that man has ever built?

Re:How Many Times?

[dangitman (862676)]

Or do you recommend using a not even fully designed, coded, and tested language for controlling the most complex piece of equipment that man has ever built?

They put computers in bra straps now? Sheesh, I was just getting used to the old ones, and now this?

Re:How Many Times?

[Draknor (745036)]

I thought five 9's was the standard for uptime/availability. And you don't achieve that by having 1 server that has five 9's, you achieve that by having a group of servers, so when one goes down, your service is still up.

Space shuttle's a little different:

http://www.fastcompany.com/online/06/writestuff.ht ml [fastcompany.com]

Here we're talking *six* 9s of *bug-free code* (1 error in 420,000 lines of code in the previous version). Not uptime -- bugs. Mistakes. For the simple reason that if you make a mistake on the Shuttle, people die.

You won't get a Java implementation that bug-free without a crack team of developers working for decades, by which point Java will be just as outdated then as the Shuttle code is now.

Remember -- if it ain't broke, don't fix it!

Re:Bites me

[clickclickdrone (964164)]

>capability of missiles (and other defence systems) to handle war through a year-end changeover?
That's why every new year all the soldiers climb out the trenches, swap chocolates, cigarettes etc, and shake hands before climbing back in and resuming war the next day.