Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
NASA Space Science

NASA Loses Contact With Space Station Over Software Update 111

Posted by Soulskill from the i-didn't-know-they-used-itunes dept.
kodiaktau writes "Reports early Tuesday morning say that a software update to the International Space Station caused a communications blackout with Houston control. Remediation of the update has allowed the astronauts limited communication every 90 minutes or so. It is expected that the issue will be resolved today."
This discussion has been archived. No new comments can be posted.

NASA Loses Contact With Space Station Over Software Update More

NASA Loses Contact With Space Station Over Software Update

Comments Filter:

  • Should've read the manuals. (Score:5, Funny)

    by ZaMoose ( 24734 ) on Tuesday February 19, 2013 @03:10PM (#42947755)

    I mean, even my Linksys warns me to only update firmware when I've got an Ethernet cable plugged in to it, because you know how wireless upgrades go.

    • Re:Should've read the manuals. (Score:5, Funny)

      by eksith ( 2776419 ) on Tuesday February 19, 2013 @03:12PM (#42947777) Homepage
      They don't make CAT6a cables that long :/

      • Re: (Score:1)

        by Anonymous Coward

        For that distance you'd need a LONGCAT6.

      • It doesn't matter as it's further than 100 meters away.

        • Re: (Score:3)

          by tlhIngan ( 30335 )

          It doesn't matter as it's further than 100 meters away.

          That limit is because of how long it takes for a minimum sized packet to traverse the network for collision detection. If you can guarantee there won't be collisions (a switch does this) then you're pretty much not limited by length anymore other than signal strength. All you need is a bunch of switches to boost the signal up and you're golden.

          Back in ye olde days of hubs, they were half duplex because the transmitting NIC would send bits down the line

        • Well then, use several cables that are shorter than 100 meters.....
          Or a very long cat.

    • Well, apparently, they don't go *up* very well, anyway...

  • What could possibly go wrong? (Score:5, Informative)

    by IronHalik ( 1568993 ) on Tuesday February 19, 2013 @03:11PM (#42947763)
    http://imgur.com/gallery/fv2rg9a [imgur.com]

  • The rumor has it... (Score:5, Funny)

    by toxygen01 ( 901511 ) on Tuesday February 19, 2013 @03:15PM (#42947805) Journal
    that it was Java update... what to say...

  • Windows 8 (Score:5, Funny)

    by puddingebola ( 2036796 ) on Tuesday February 19, 2013 @03:16PM (#42947813) Journal
    Actually, it was just some confusion when someone right swiped on Windows 8. The Charms bar came up and then the weather app launched and nobody knew how to close either one.

  • Communication Reestablished (Score:5, Informative)

    by NoKaOi ( 1415755 ) on Tuesday February 19, 2013 @03:19PM (#42947855)

    http://www.space.com/19853-space-station-contact-lost-nasa.html [space.com]

    Update: NASA has reestablished contact with the International Space Station. For the latest news, read: NASA Restores Contact with Space Station

    As far as NASA officials can tell, the space station's loss of communications was unrelated to the software update, Kelly Humphries, a public affairs specialist at NASA told SPACE.com. It was a coincidence that the space agency lost contact with the station as the computers were being updated.

    • Coincidence is really a stretch. How many times have I had to support someone with their problems (PC, DVR, Car, etc.) where they claim nothing changed nothing changed.

      Later to find out "Oh yeh, I DID just install something before the problem... I just thought it was a coincidence"

      There was a quote from a TV series called "The Unit" which accurately depicts my thoughts on the matter.
      Mack: You believe in coincidence?
      Bob: Do you?
      Mack: Like I believe in God, I believe in it. But I've never seen it.

  • Yep (Score:5, Insightful)

    by roc97007 ( 608802 ) on Tuesday February 19, 2013 @03:20PM (#42947861) Journal

    Every time the offshore admins want to apply an update, I ask them "what is your contingency plan should you brick the server?" and they always answer "Call the vendor". Sigh.

    • Re:Yep (Score:5, Interesting)

      by omglolbah ( 731566 ) on Tuesday February 19, 2013 @03:30PM (#42947983)

      We do a full image backup of the server.
      Then we shut it down (they're all redundant) and remove one set of drives from the mirrored raid.
      Start back up.
      Run the update.
      Verify that the update went ok
      Perform new image backups.
      When everyone is satisfied shove the mirrored drives back in.

      Then again, we're "offshore" as in an oil rig and patching control system HMI servers... so I guess having a contingency plan would be required. This rig (where I am at now :p) makes 50 million USD a day in natural gas.. so uptime is paramount!

      • Re:Yep (Score:5, Insightful)

        by roc97007 ( 608802 ) on Tuesday February 19, 2013 @03:41PM (#42948107) Journal

        That's a different, and probably more appropriate definition of "offshore". That also sounds like a very good procedure. I would have added "we only upgrade firmware to solve specific problems and vulnerabilities appropriate to our environment, not just because it's 'the latest'". But I'm told I'm too conservative.

        Here, the admins are offshore (as in, physically on the other side of the world) but the machines are still local. They've rebadged former mainframe operators to be "hands and eyes" in case a button has to be pushed or a memory stick changed out. Of course, "hands and eyes" have had no hardware training whatsoever. I made some fuss recently when I caught one of them changing out a memory stick with the gator clip on their wrist strap dangling in the air. The response was to raise the issue as to why I still had access to the computer room? Geh.

        • Offtopic, sort of, but never have I seen, in about 8 years or so, a computer component being toasted by static discharge, and here nobody ever uses grounding when they work with hardware.

          • Comment removed based on user account deletion

          • Re: (Score:3)

            by asc99c ( 938635 )

            Neither have I, and I don't bother with any precautions when working on PCs. But I have also assembled an AIX server which would have been just about into six figures in US dollars, and thought for that one I'd spend five minutes to go and find the anti-static equipment!

          • When you're handling 15000 dollar circuit boards using a wrist-strap is always a good idea..

            Not much protection of the circuitry here: http://i01.i.aliimg.com/img/pb/349/310/467/467310349_361.jpg [aliimg.com]

            Some of the input slave cards are highly sensitive to ESD, and we've had some fry.

            Damn reliable if properly handled though. Do not think I have ever seen one fail in use.

          • Offtopic, sort of, but never have I seen, in about 8 years or so, a computer component being toasted by static discharge, and here nobody ever uses grounding when they work with hardware.

            There are ways to handle computer components safely without a grounding strap. For instance, putting bare forearms against the chassis frame when removing or inserting components.

            One could argue that the ground strap is there to insure that you are always statically grounded even if you forget, or don't know about, other precautionary measures. I'd like to posit that someone with absolutely no hardware training probably doesn't know to touch the chassis before handling components, but apparently this also

        • You and I must work for the same big company. How many times have I notified someone about something that is incorrect, and instead of expected reaction "right, it's not supposed to be that way" I got "you're not supposed to know about it".

      • Re: (Score:1)

        by Anonymous Coward

        Serious question (unfortunately posting anonymously to not lose some moderation) ...

        Its 2013... any reason you're not using virtualization for these sort of things? It may be overkill for abilities like live migration, particularly if your software is intrinsically highly available and bringing a server down doesn't matter, but a snapshot revert is always going to be the fastest way to recover in an emergency.

        • Large app servers, database back-ends, and outward facing servers remain physical. Just about everything else is a candidate for virtualization. It so happens that my responsibilities are primarily with big physical servers.

          But let's talk virtualization... you haven't seen a mess until you've seen a firmware update pushed out that takes three-quarters of your VM farm offline. Virtualization allows you to be responsible, by migrating off a few servers at a time, upgrading them, and if successful, move the

      • "This rig (where I am at now :p) makes 50 million USD a day in natural gas.. so uptime is paramount!"

        If it's making that much, then natural gas is too expensive, almost by definition.

        • Probably means gross revenue. I'm sure it takes a few dollars to run the rig. Companies make money. Get over it.
          • Yeah. Gross revenue is a completely different matter. But that also means it doesn't "make" quite that much, that's just how much it takes in.

        • The Troll A platform can produce 122 million standard cubic meters of gas in a 24 hour period.
          That is 4.3 billion cubic feet of gas per day.

          • You're just reinforcing my point.

            If one rig can produce enough to "make" $50 million a day, then the retail price is far too high.

            But if, as the other poster mentioned, that is actually gross revenue rather than profit, it's a bit of a different story. I'd still be interested in what kind of margin they're operating on.

            • If one rig can produce enough to "make" $50 million a day, then the retail price is far too high.

              How would you go about working out what a reasonable figure is?

              • "How would you go about working out what a reasonable figure is?"

                That's a very good question. Normally, I'd say let the free market sort it out. But oil and gas are fraught with such an enormous hodgepodge of government regulation, taxation, and subsidies that it is pretty hard to claim anything like a "free market" even exists for those things.

                • That's a very good question. Normally, I'd say let the free market sort it out. But oil and gas are fraught with such an enormous hodgepodge of government regulation, taxation, and subsidies that it is pretty hard to claim anything like a "free market" even exists for those things.

                  Then how do you decide it's "too high" in the first place?

                  • Because at a glance (just off-the-cuff estimating, I make no claim to any kind of thorough analysis), if ONE RIG can make $50 million profit in a day (the word "make" was used), then multiply that by the number of rigs, and you start to get pretty huge numbers.

                    What I was trying to say is that based on my (admittedly lacking some data) estimate, this would have to lead to prices that are far higher than what a free market would set.

                    But that's all pretty theoretical, since I don't have all the numbers,

    • Re: (Score:3)

      by AmiMoJo ( 196126 ) *

      I knew an admin who was so scared of updates breaking something and getting called up to London at 6 PM to fix it he just never bothered. Every time the staff infected the RDP server with a virus because they were stuck with IE7 he blamed them, and one even lost her job over it.

      Eventually the company got fed up, ditched us and found another support company. The first thing they did was install all the updates and virtualize all the servers.

      • I knew an admin who was so scared of updates breaking something and getting called up to London at 6 PM to fix it he just never bothered. Every time the staff infected the RDP server with a virus because they were stuck with IE7 he blamed them, and one even lost her job over it.

        Eventually the company got fed up, ditched us and found another support company. The first thing they did was install all the updates and virtualize all the servers.

        Right, but there is a huge space between "being cautious about updates" and "never installing and update". System updates, and most especially firmware updates need to be staged just like any application update. You apply them to non-critical systems first, test, and then cautiously apply them to up the line to your most critical system. Even then, you have to watch your application support matrix. We've had issues where drive-by updates (installed over the weekend by offshore admins) brought us to an u

  • Hello, IT (Score:5, Funny)

    by Clancie ( 678344 ) on Tuesday February 19, 2013 @03:21PM (#42947867)
    Have you tried turning it off and on again?

  • I don't think they update on the fly they send the (Score:3)

    by Joe_Dragon ( 2206452 ) on Tuesday February 19, 2013 @03:23PM (#42947895)

    I don't think they update on the fly they send the files and verify that they are 100% before starting a install locally

  • Don't worry, it's not bricked. (Score:5, Funny)

    by Nanoda ( 591299 ) on Tuesday February 19, 2013 @03:25PM (#42947911)

    Someone is going to have to hold down the button on the side for 10 seconds though.

  • So, the entire space station only has *one* single radio communications device, with no redundant/emergency backup?

    • They also have an amateur radio station on board that can be used for emergency communications. It would have been a CB, but West Virginia couldn't get their mobile home module into orbit.
      • As an amatuer radio operator, I'd love to take up the slack and do comms for them. Get enough of us, and we can provide global support, free of charge, and would be honored to do it! There *may* be some coverage holes in the south Pacific...
        • Kwaj could pickup the South Pacific. There were plenty hams when I was out there... But the problem is NASA, to quote one of my prior NASA co-worker (one of the few actual skilled) "NASA's welfare for nerds...". NASA and its contractors are in serious need of a "job skills" audit, at least in IT. It's an embarrassing living, breathing, self-perpetuating Charlie Foxtrot.
          • I was just asked (privately) to be more specific, which I found interesting. So it'd be a worthy idea to start an audit, if such an audit existed, with the NISN IT Security group based out of Marshall Space Flight Center primarily outsourced to SAIC. That should get the ball rolling AND KEEP IT ROLLING.
            • They have Sr. Computer Scientists pulling deep six figures (of our tax dollars) that don't know the difference between 8086 (x86) and RISC, or how to change the IP address on a windows box and that stared at a shell prompt like it was a practical joke. I later found out that she was previously a bar tender "friend" of the civil servant who called in a favor or something...no wait, that was a different one.

              • Dude. Stop. Just stop. You're depressing me.

                Six figures? SIX FIGURES?? Goddamnit, I need me some of THAT welfare.

                • Yeup, actual relevant technical skills is irrelevant. Very disconcerting...especially considering that there are MANY out of work Sr. Computer Scientists with true, impeccable professional and schollastic pedigree. It wouldn't be as bad if those people counted their blessings and made (or even faked) the slightest effort to become qualified for their positions. Instead they flaunt their political prowess and attend only to that which expands their primary technical skill, "who they know". The foundation

  • Talk with the crew on Wednesday (Score:1)

    by Anonymous Coward

    On Wednesday Feb. 20, 150 of NASA's social media followers and their guests will have the unique opportunity to talk to three of the six crew members aboard the International Space Station, and speak with agency scientists and engineers about the ground-breaking research taking place daily on the orbiting laboratory...

    http://www.nasa.gov/connect/social/social_ISSscience_feb2013.html [nasa.gov]

  • I guess I'm not the only one having problems with the xbox live update.

  • Screen is stuck on this: (Score:4, Funny)

    by Bill Hayden ( 649193 ) on Tuesday February 19, 2013 @03:32PM (#42947999) Homepage
    Applying update 8 of 27...

    Please do not turn off your computer.

  • No one ever goes to the source (Score:2, Insightful)

    by Anonymous Coward

    This has been fixed for over three hours.

    http://www.nasa.gov/mission_pages/station/expeditions/expedition34/e34_021913.html

    Is it really that hard to check with NASA?

    • Re: (Score:3)

      by PRMan ( 959735 )
      Are you kidding? I'm just happy when Slashdot isn't 3 DAYS behind. 3 hours is a massive improvement.

  • Ha!

    Change is bad!

    Nothing good ever comes from it.

  • In Space... (Score:1)

    by Anonymous Coward

    No one can hear when you BSOD

  • What is that, once an orbit? Does this mean they lost routing from other stations, and can only communicate when over US receivers?

  • This is what happens when you patronize a phone vendor who locks the bootloader.

  • Software Verification and Validation (Score:5, Interesting)

    by DERoss ( 1919496 ) on Tuesday February 19, 2013 @05:53PM (#42949779)

    I spent over 20 years of my career (now retired) working for a company that did independent verification and validation (IV&V) of software used by the military to operate its unmanned space satellites. Not once was a satellite lost from an error in the software if we were involved.

    There were some 10 or more other, unrelated companies developing software for various space satellites. We did more than merely test the resulting products. We started by reviewing the developers' design documents; our reviews required responses or revisions before any coding could occur. Next we reviewed the developers' programming documents; our reviews required responses or revisions before programming could be completed. Then we reviewed the developers' test documents; our reviews required responses or revisions before the developers could conduct their own internal unit tests. We attended the conduct of those internal tests and audited the results to ensure that the purposes and criteria of the tests were satisfied.

    Finally, the developers would deliver their software to us. We would test the products at the package and system level. We looked at how products from different developers interfaced with each other, whether human interfaces were reasonable, and whether the government's requirements had been met. Our test documents were reviewed by the military organizations that would be using the software, and we did not start testing until we responded or revised our test documents.

    This IV&V process approximately doubled the cost of providing software. However, no such software caused a satellite to land on the White House or (worse) on the Kremlin. In the early 1990s, the Pentagon decided to save money by eliminating IV&V. I continued testing software for military satellites, but then it was within the companies that developed the software. When schedules or costs were at risk, testing was cut short.

    Sic transit gloria mundi.

    • My stint on the ISS testing team years and years ago (back when the Interim Control Module looked like it was needed) showed that, though everyone wanted the thing to work, there were more pressures to make papertrails than make quality work. That being said, oh man, that Ku-Band data stream got seriously hosed.

    • Do you have any figures on satellite losses after IV&V was stopped?

  • Well, if the Kenwood TM-D700A is back on the air (ie, after -its- recent glitch),
    Ham Radio could save the day, albeit only when there's another Ham operator
    on the ground, in the coverage footprint to talk to & (hopefully) reliably relay
    messages to / from NASA.

    Ideally, the very-speed (9.6 Kb/Sec, AX.25 data mode) packet radio link
    would let ISS forward its messages into the world-wide store-and-forward
    network, as well as fetch any incoming messages.

    (Of course, the same radio provides an unencrypted voice

  • I worked at NASA for 4 years, there are some talented people but they're so far and few between those people must hide their skills like early Christians hid their beliefs. It was shocking...it's a miricle more and much worse doesn't happen on a regular basis.

Slashdot Top Deals

In the long run, every program becomes rococco, and then rubble. -- Alan Perlis

Close