NASA Loses Contact With Space Station Over Software Update 111
kodiaktau writes "Reports early Tuesday morning say that a software update to the International Space Station caused a communications blackout with Houston control. Remediation of the update has allowed the astronauts limited communication every 90 minutes or so. It is expected that the issue will be resolved today."
Re:Yep (Score:5, Interesting)
We do a full image backup of the server.
Then we shut it down (they're all redundant) and remove one set of drives from the mirrored raid.
Start back up.
Run the update.
Verify that the update went ok
Perform new image backups.
When everyone is satisfied shove the mirrored drives back in.
Then again, we're "offshore" as in an oil rig and patching control system HMI servers... so I guess having a contingency plan would be required. This rig (where I am at now :p) makes 50 million USD a day in natural gas.. so uptime is paramount!
Software Verification and Validation (Score:5, Interesting)
I spent over 20 years of my career (now retired) working for a company that did independent verification and validation (IV&V) of software used by the military to operate its unmanned space satellites. Not once was a satellite lost from an error in the software if we were involved.
There were some 10 or more other, unrelated companies developing software for various space satellites. We did more than merely test the resulting products. We started by reviewing the developers' design documents; our reviews required responses or revisions before any coding could occur. Next we reviewed the developers' programming documents; our reviews required responses or revisions before programming could be completed. Then we reviewed the developers' test documents; our reviews required responses or revisions before the developers could conduct their own internal unit tests. We attended the conduct of those internal tests and audited the results to ensure that the purposes and criteria of the tests were satisfied.
Finally, the developers would deliver their software to us. We would test the products at the package and system level. We looked at how products from different developers interfaced with each other, whether human interfaces were reasonable, and whether the government's requirements had been met. Our test documents were reviewed by the military organizations that would be using the software, and we did not start testing until we responded or revised our test documents.
This IV&V process approximately doubled the cost of providing software. However, no such software caused a satellite to land on the White House or (worse) on the Kremlin. In the early 1990s, the Pentagon decided to save money by eliminating IV&V. I continued testing software for military satellites, but then it was within the companies that developed the software. When schedules or costs were at risk, testing was cut short.
Sic transit gloria mundi.