Why Hubble Broke and How It Was Fixed 73
angry tapir writes "I recently had the opportunity to sit down with Charles (Charlie) Pellerin, who was NASA's director of astrophysics when the Hubble Space Telescope launched with its seemingly fatally flawed optical system. Pellerin went on to head up the servicing mission that finally fixed the telescope and for that was awarded NASA's highest honor, a Distinguished Service Medal. Since Hubble he has done a lot of thinking about the problems that led up to the error and how organizations can best avoid making similar mistakes."
Afraid to speak up about problems (Score:5, Insightful)
The article mentions that the contractor was afraid to bring up problems.
That, plus the mentality from management that people who bring up problems are "troublemakers," "negative," "not team players," etc. (because they've put too much of their ego or political capital into a project) has got to be responsible for more disasters, large and small, than any other deadly combination.
I worked for a large nonprofit that blew money on doomed projects as though money grew on trees. Each time, it started with somebody, usually a contractor or somebody else who stood to gain from it, flattering the leadership that this was huge and visionary and would make or save them millions. Then the organizational mind control started, where everybody was saying that it was the greatest thing ever. Then the flawed project management started. Then when the cracks were obvious, people who pointed them out were vilified as naysayers. It was only the lower-downs who said anything because to rise, one had to be a "team player," and the organization was hierarchical enough that lower-downs were ignored. Then denial that there were problems, together with tossing more money at it (including adding more people to a software project at the last minute because that always works). Then even when the leadership [sic] team [sic] all realized there were problems, they all waited until the person responsible for the project was willing to concede defeat. because in a political environment, nobody wants to confront somebody who might retaliate
Those elements are the inevitable recipe for disaster for any project, but it's fear that drives virtually all of them. Fear of not looking good (note that the Congresscritter didn't yell about wasting taxpayer money, she yelled about being made to look bad), loss aversion, fear of admitting a mistake, fear of speaking up.
Pellerin was brave enough to do something technically illegal and scrape up the funds for servicing it.
That is what a leader does.
Re:The real story... (Score:5, Insightful)
It's not only the $50k for the test. Most likely there will millions in cuts and this test happened to be in the mix. It would be nice if you only had to pay for the tests that showed problems. It would make engineering much easier. Unfortunately you have to test for everything even the stuff that works fine.
Re:Interesting read (Score:3, Insightful)
Re:The real story... (Score:5, Insightful)
Your anecdotal story is intersting, but it fits right into what he was talking about with the Management failures at NASA. Clearly it wasn't the lack of that test that caused the problem. It was a management decision to not perform it. Probably under the exact same pressures. Even if it had been performed though, who's to say they wouldn't have rationalized away the results like they did all the other failed tests?
"We tested that mirror over and over and over with a different kind of device, the old style refractive null corrector," Pellerin says. The results? "Half wave of error, half wave of error, half wave of error." "So some people sat down and said, 'What's going on?" Pellerin recalls. "The mindset was that the mirror can not be other than perfect. So something else is happening. They concluded that the mirror was sagging under the force of gravity in the three point mount rather than being on the bed of nails by half a wave. "Well it turned out that was wrong. But they rationalised, rationalised, rationalised.
...
The project had suffered other challenges beyond fabricating and mounting the mirror; staff were being "hammered" all the time, Pellerin says. In addition there was constant angst about how far the project had gone over budget. "Hubble's initial budget was $434 million we closed it at $1.8 billion just for the flight segment; big overruns." "So the way it works is you tend to blame the people doing the work," Pellerin says. "So we're hammering on them, hammering on them so they had no free time or no inclination to track down anything that wasn't a critical problem because we have other critical problems. Difficult technical things that we couldn't solve yet." The review board also found that a hostile environment had been created for the contactor, which meant "they told us about any problem at their peril," Pellerin says.
Re:Interesting read (Score:5, Insightful)
You should read the article, because your comment is exactly the kind of thing he is talking about. Technical people who think they have found a technical problem, therefore the solution is to correct that problem. If the problem was really that measurement system the US uses is 'wrong', then how can the US have so many successful space missions? The problem is not that there are multiple measurement systems, or that one is somehow superior to the other. The problem is that the teams did not communicate successfully - not a technical problem at all. And don't say 'well, if the stupid US would use the same system as the rest of the world it wouldn't be a problem', because that just shows you completely missed the point. The point is that there was ineffective communication - a leadership problem - not simply a technical problem.
Re:The real hero (Score:4, Insightful)
Looks like you did not read the article, but this is slashdot after all.
He talks about teamwork. Individuals contribute, but group dynamics are very important, and perhaps a deserve larger share of the success than any individual.
By picking someone out as the hero you are committing the same errors as in the past.
The "anti-indiviudal abilities agenda" (Score:3, Insightful)
The anti-individual abilities agenda is routinely promoted by managers, project managers and other people engaged in the management layers (management consultants, business schools etc.). The motive is pretty clear: Many bosses don't like admitting that the success of their project comes down to individual abilities of a few core members on the project. After all, what is the value of management then, they ask? It's like the tail wagging the dog.
However, this is just denying reality. I can firmly say that on any project of major size I worked on, the was a few 5-10% of people on the project running the show. This in itself is not very surprising, what is surprising is the fact that these 5-10% were not centered at the top of the pyramid. Rather, it was evenly spread out over all 'layers' from 'highest to lowest'. These people (by virtue of their skills and dedication to the project, something that is often lacking with the project management itself!) automatically assume a role of authorities whether management likes it or not. It's simply the only way to get things done. Let's face it, on any project there's going to be a lot of 9-5'ers that don't really care. They are never the ones driving the car, nor should they. It's the 5-10% who has both the ability to and the interest in getting the job done that counts. Those that dream about the project at night and who feel their personal honour is at stake in making it succeed. Also, as Fred Brooks noted in 'a mythical man month', some (sub)projects are like surgery. You need one highly skilled person to be in charge and carry out the job, and the rest of the team members are really just accessories of that person. Their contribution can be important of course, but at the end of the day, all choices, responsibility resides with the 'surgeon' etc.
I think the lesson to be learned from these observations is that management needs to accept that this is the structure that projects will generally fall into, no matter what they do. The job of management is to get the best result out of it. On projects with poor management that creates obstacles for progress and makes lots of bad choices (this often happens on politically infested projects as well as on projects where management doesn't have a clue about the technical aspects), often the project finds a way to completely bypass management. Decisions by management may be outright ignored, or important decisions are never brought up to this level but are just made behind the scenes. This is a very dangerous situation since important decisions may not be properly reviewed and may not even be known by all stake-holders. While most decisions taken may have been correct, it takes just one bad decision to jeopardize the project, and problems related to this kind of "skunkworks decisions" tend to surface very late where they may cause huge problems, sometimes disasters.
The job of management is to embrace the individual abilities, and to listen carefully (but of course not uncritically) to arguments brought forward, no matter if it is from a project manager or a "lowly" techie. They need to make a decent effort to try to understand what they are talking about, even if the explanations are not always clear and even if it can sometimes be highly technical.
Re:Interesting read (Score:4, Insightful)
It's an important distinction because the same error can happen even if you work entirely in SI units. If I write down a number in kilonewtons but fail to write down the units, and you assume it is just newtons, we end up with the same problem. I've seen this happen countless times in the lab and while tutoring, with kids plugging grams into an equation when they're supposed to be using kg. (Which BTW is one stupid thing about SI units - really confuses the kids that the base unit for everything else has no prefix, but the base unit for mass is a kilo-gram.) Fortunately, forcing them to write down the units after every number usually takes care of this problem.
In science and engineering, any time you see a number without units, your immediate reaction should be to ask the person who provided the numbers what the units are. (Actually you should be ripping him a new one for failing to write down the units, dimensionless numbers excepted.) Never assume the units, always ask.