Visualizing False Positives In Broad Screening 365
AlejoHausner writes "To find one terrorist in 3000 people, using a screen that works 90% of the time, you'll end up detaining 300 people, one of whom might be your target. A BBC article asks for an effective way to communicate this clearly. 'Screening for HIV with 99.9% accuracy? Switch it around. Think also about screening the millions of non-HIV people and being wrong about one person in every 1,000.' The problem is important in any area where a less-than-perfect screen is used to detect a rare event in a population. As a recent NYTimes story notes, widespread screening for cancers (except for maybe colon cancer) does more harm than good. How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?"
Second opinion (Score:3, Informative)
While it's true that there will be false positives, as well as false negatives, you don't convict someone, or have a lung removed, without further testing. When I was diagnosed for cancer, I was tested and re-tested to verify that there was, indeed, cancer. The same goes with screening for terrorists, or anything else. Did the article mention the rate for false negatives as well? After all, if you have a five pound tumor hanging off you face, and your doctor tells you there's nothing wrong, I'd definitely want a second opinion!
Re:Math ftl (Score:5, Informative)
A box (Score:5, Informative)
Back during the TQM fad they'd make this point by giving everyone a clear plastic box with 10,000 little balls in it. There was a cribbage board like affair in it, with 1,000 holes, such that by inverting and shaking the box, then turning it upright, 1,000 of the balls would settle into the holes more or less at random, but still be visible through the clear box. The balls were color coded -- 10 red balls, 40 black ones, 50 blue ones, and the rest white. The odds of getting no red and no black are lower than 1%, contrary to most people's expectations.
This was used to drive home a point about the difficulty of "testing in quality" (quality tests suffer false negatives and if there are, say, 1000 such individual measurements on a piece of machinery it's nearly impossible to ship a machine without at least one thing wrong unless the tolerances are well controlled at the point of manufacture). The same idea works any time you want to illustrate the effects of low-incidence events on a large population.
I've always wondered how much injustice is perpetrated by drug screening on large populations, since false positives do occur and statistically must occur twice in a row at least some of the time, which is the threshold considered conclusive proof of abuse by most employers and the courts.
Re:Math ftl (Score:3, Informative)
Not math, interpretation (Score:4, Informative)
"Works 90% of the time" here means that it will correctly identify a person as terrorist or not-terrorist in 90% of tests.
On a sample of 3000 with accuracy 90%, you will end up with 300 results guaranteed to be wrong or ambiguous, which may or may NOT mean the subject is a terrorist. To be safe, obviously you have to detain these "ambiguous" subjects.
Considering that we know the number of terrorists is incredibly small (from a UK perspective, I'd say something like 100 in 70 millions, or 1 in 700.000, probably even less), we can deduce that these tools are guaranteed to victimize thousands of innocents (at least 69.999) for each "terrorist" ever caught.
Re:using a screen that works 90% of the time (Score:4, Informative)
And here, silly statisticians use two numbers, alpha and beta to represent failure rates. Someone needs to educate them that they really only need one number
Infographics to the rescue (Score:3, Informative)
Draw a picture. People's visual intelligence is much higher than their literary/verbal intelligence. Descriptions in words are difficult to understand when the meaning of the words being used is not clear, uses domain specific jargon (such as 90% accuracy in relation to statistics) and especially when it requires that the recipient of knowledge perform a mental calculation or solve a mental equation.
An effective picture would be one of a thousand people (stick figures or silhouettes will do) with 10 positioned in front. A caption over the 900 in the big group would say "Tested Negative (These people are NOT Terrorists), the caption under the 10 in front would say "Tested Positive (These people may or may not be Terrorists - We don't know)".
Then ask people how they would feel if they were in the group of 10 and were going to be shipped off to a military holding cell to await further investigations.
Nice article. -ish (Score:5, Informative)
I think they totally forget that there is ALSO a 10% possibility that you _don't_ detect the terrorist...
Watch this TED : http://www.ted.com/talks/peter_donnelly_shows_how_stats_fool_juries.html [ted.com]
Re:Understanding efficiency (Score:3, Informative)
Please do not simply press the "9" key until you get bored when it is more readable and more accurate to use words like "very accurate".
For example, it would be difficult to empirically measure the stated accuracy of your test, since it's inaccurate 1 time in 100 billion.
This message has been brought to you by the Society for the Elimination of Superfluous Quantification.
Re:Math ftl (Score:3, Informative)
Re:A box (Score:4, Informative)
I've always wondered how much injustice is perpetrated by drug screening on large populations, since false positives do occur and statistically must occur twice in a row at least some of the time, which is the threshold considered conclusive proof of abuse by most employers and the courts.
this very problem came up in England a while ago, but for SIDS deaths. If I recall correctly, some statistician testified at a murder trial as to the infinitesimal chance that a mother would have two infants die from SIDS separately. In fact, granted the size of the population, it is not unlikely for two SIDS deaths to happen to some mother somewhere in the country. Perhaps someone else can remember the details. here's an article but it's not free: http://bjps.oxfordjournals.org/cgi/content/abstract/axm015v1 [oxfordjournals.org]
Re:Come On (Score:5, Informative)
Turbans are worn by Sikhs. This is a completely different religion to Islam which is alleged to harbour these terrorists.
Re:Math ftl (Score:4, Informative)
You are wrong. Lets suppose there are 60,000,010 people in the country, of which 10 are terrorists and 60,000,000 are not terrorists.
The test will incorrectly identify 600,000 of the non-terrorists as terrorists, and 1 of the terrorists as a non-terrorists.
What this means is that out of the 600,009 people it identifies as terrorists, only 9 actually are.
Re:Come On (Score:3, Informative)
The 9/11 and 7/7 attacks etc were instigated by self proclaimed Muslims. There's no "alleged" about it, it's just a fact. Note that I don't believe that all Muslims are terrorists, or that all terrorists are Muslim, that's just stupid. But the most widely publicised and recent terrorist attacks have been strongly linked with Islam. That doesn't mean that Islam "harbours" terrorists, a lot of Muslims don't agree with these violent attacks. But it's pretty safe to say that the majority of attempted terrorist attacks within the next few years are likely to be instigated by Muslim fanatic groups who are pissed off about the whole invasion situation in Aghanistan/Iraq/Iran. The 7/7 attacks here in the UK were "carried out by 4 British Muslim men who were motivated by Britain's involvement in the Iraq War" according to Wikipedia.
Re:Think you understand these things? Try this... (Score:3, Informative)
A family with two children is chosen at random from a large population.
If I tell you only that they have at least one daughter, what is the probability that both children are girls?
Most people can get that one (it's 1/3), but fail miserably on this question:
You are incorrect. Your statistic would be true if we were randomly picking family with two children until we came across one with (at least) one girl. There's a 1/3 odds there we'd pick one with two girls, and 2/3 that we'd pick one with just one.
However, that is not what you said. You said we picked the groups at random, and, hence, telling us the gender of a child tells us nothing about the other one. The genders are entirely independent of each other.
You can see how that works by imagining that the second child has not actually been born yet.
Or imagine it as coin flips. If I announce the result of one coin flip, it's not going to alter the other. If I make pairs of coin flips, and deliberately select a pair that has at least one tails in it, however, I have removed certain flips from the odds.
You actually understand this in your second example, and get the right odds, but surreally miss it in the first, despite using exactly the same example. If only girls are named Mary, saying one is named Mary is exactly identical to saying one is a girl. Your two examples are the same. You meant for your first example to be:
If we pick out two parents who have at least one girl, what are the odds that their other child is a girl?
That has the odds of 1/3, because the possibilities are M/F, F/M, and F/F.
Re:Come On (Score:3, Informative)
What a load of crap, back in 2006 NBC dateline had a bunch of muslims go to a Nascar race and see if they were harassed, guess what they were NOT bothered at all.
Funny, I can't find any record of Dateline actually running that segment. What I do find is a billion news articles about how NASCAR and others like Michelle Malkin got their panties in a twist about it with the typical faux indignation of the bigotted right. I would have expected NASCAR's PR people to be smarter than that, but apparently not.
This sort of idiotic bollocks is what perpetuates the myth that the US is full of racists.
You've picked a strawman. Just because such attacks do happen does not mean that "the US is full of racists" what it does mean is that there are some racists here. Don't pretend that just because your silly strawman is false that no such racist attacks happen at all.
http://ibnlive.in.com/news/sikh-attacked-in-another-hate-crime-in-new-york/57501-3.html [in.com]
http://www.nypost.com/seven/09162007/news/regionalnews/muslim_biz_gal_beaten.htm [nypost.com]