Metrics Mania and the Countless Counting Problem 138
mobkarma writes "Einstein once said, 'Not everything that can be counted counts, and not everything that counts can be counted.' A New York Times article suggests that unless we know how things are counted, we don't know if it's wise to count on the numbers. The problem isn't with statistical tests themselves, but with what we do before and after we run them. If a person starts drinking day in and day out after a cancer diagnosis and dies from acute cirrhosis, did he kill himself? The answers to such questions significantly affect the count."
cirrhosis (Score:3, Insightful)
Re:How about this (Score:5, Insightful)
I don't disagree, but, it cuts both ways. I think the article has a point... the numbers only have meaning in context.
If I tell you "X people die every year from being shot, in their home, with their own gun", that tells you something. It evokes images of burglars or irresponsible people playing. It SOUNDS like a statement about how safe or dangerous it is to have a gun in your own house.
However, if my number "X" includes suicides, well, then how much of a statement about the relative danger of owning a gun am I making? How about if I can find no link between owning a gun and committing suicide?
Clearly the statement is correct, "shot, in their home, with their own gun" but, even so, its misleading if you then use the numbers wrong.
Take texting while driving. The claim is 900 deaths a year. How do they come at that number? Even better, how big is that number? 900 sounds like a lot.. However... its less than the estimated number of serial murder victims in the US. Overall driving deaths are more like 40,000 a year. Context is everything. If I said "about 1 driving death in 40 is related to txting while driving" thats suddenly a lot smaller, yet, represents the same data.
frankly, I tend to think a LOT of statistics are meaningless. NY state enacted a law against handhed phone use while driving. It resulted in a 70% decrease in OBSERVED use. There was no decrease at all in accident rate.
What this tells me is, someone really believed that this was going to make a difference, came up with numbers and statistics and, in reality, the one little item that he picked out had about as much bearing on accident rates as the price of butter in bangladesh does.
Much of the time statistics are used to just bullshit and make it look like we aren't playing blindfolded darts when we make public policy.
-Steve
Goodhart's law (Score:3, Insightful)
Sounds like a restatement of the simultaneously-discovered Goodhart's Law [lesswrong.com], Lucas critique [wikipedia.org], and Campbell's Law [wikipedia.org].
Basically, once you start measuring something as a proxy for what you really want to know, people start to take the proxy into account when making decisions, to the point where it becomes useless as a measure for whatever it was intended.
Here, people take these cancer tests as a measure of their probability of cancer. But once they start to treat them as reliable, they start doing more self-destructive things, destroying the correlation between the proxy (the cancer test) and the actual probability of cancer.
Re:How about this (Score:5, Insightful)
I'd wager that the 1 in 40 people who've died from texting while driving came out of a sample of 1 in 40 drivers who don't put enough importance into paying attention to the road. So, take away texting drivers, and you'll still have 1 in 40 people dying because they were adjusting their radio one station at a time without looking up, or rolling up the rear-passenger window by hand because they don't have power windows.
I don't think texting while driving has increased accidents, I just think it's made it easier to point out who the stupid drivers are.
Lies, damned lies, and statistics (Score:5, Insightful)
Many years ago, I had an in-depth discussion about gathering statistics on heart disease with a woman on the board of the American Heart Association. This was a big deal. Serious ethical issues were in play and there was a great deal of infighting going on.
I asked her how you make a definitive decision that someone has heart disease. I was trying to figure out what to measure. Her answer surprised me. She said "You wait till they die. Then you cut out their heart and have a look." She then went on to patiently explain to me that the only thing that could be measured and evaluated were "markers" of heart disease. Those markers, as revealed by various disgnostic tests, could be mighty reliable. But you never know if someone is going to die of heart disease until they...you know...actually *die*.
Thus informed, I came to realize that what we measure is almost never what we really want to know. Measuring the right stuff is simply too hard to do. No matter where you look, this is almost universally true. In my job, for example, we fix computer problems. Thus, we measure how many incidents get closed and how much time it took. If you quickly close an incident, then surely you've provided good service, right? Most slashdotters should realize that's not true. In fact, my job is actually to get other, more important workers back to work asap. The only way to measure that would be to interview my customers and their bosses. We'd have to pry for an hour into their effectiveness to find out if I properly completed a job that took me five minutes. That's too much trouble, so we look for markers. Closed incidents. Timeliness of closures.
Measures are inadequate so often that I pretty much don't trust anything that contains them. After years of training in Quality Improvement Processes, I came to realize that the amount of time needed to understand a process and perfectly spec out what needs to be measured is 452% of the expected life cycle of the project, plus or minus a 17.5% margin of error. (Aside - How much do you trust those statistics?)
Almost no one can devote the time required to do the job (no matter what "the job" is) right. We just hope people do their best and trust to good intentions.
As a computer guy who wants things to be either "yes" or "no", unambiguously, I found this state of affairs very difficult to accept. But it's just part of being human.
Re:How about this (Score:4, Insightful)
Ah, the irony of using a statistic to prove that statistics are meaningless.
Re:Technically (Score:3, Insightful)
"Eluding". definition:
1. Evading or escaping from, as by daring, cleverness, or skill
2. Escaping the understanding or grasp of
"Alluding", definition:
Making an indirect reference
Yes, I'm a spelling nazi today....
Re:Technically (Score:5, Insightful)
Re:Technically (Score:3, Insightful)
You realize what cancer is don't you? Your cells (part of you) grow at an uncontrolled rate. It's pretty literally your body going crazy to the point where it can kill you.
Relax, it wasn't an insult or an attack. It was a word game. Enjoy.
Re:Lies, damned lies, and statistics (Score:2, Insightful)
When you quantify your work, don't forget to quantify the work you invest in quantifying your work.