Visualizing False Positives In Broad Screening 365
AlejoHausner writes "To find one terrorist in 3000 people, using a screen that works 90% of the time, you'll end up detaining 300 people, one of whom might be your target. A BBC article asks for an effective way to communicate this clearly. 'Screening for HIV with 99.9% accuracy? Switch it around. Think also about screening the millions of non-HIV people and being wrong about one person in every 1,000.' The problem is important in any area where a less-than-perfect screen is used to detect a rare event in a population. As a recent NYTimes story notes, widespread screening for cancers (except for maybe colon cancer) does more harm than good. How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?"
Simple (Score:5, Insightful)
How can this counter-intuitive fact be communicated effectively to people unschooled in statistics?
Hmm, teach them statistics?
Speech Recognition (Score:4, Insightful)
That's easy, just tell them that the screenings work about as well as speech recognition. It's 95% accurate and everyone knows how much it sucks.
Re:using a screen that works 90% of the time (Score:5, Insightful)
Exactly. So, someone who doesn't have a grasp on the terminology wants to educate folks who don't have a grasp on it either.
And this kids -- we call journalism.
Re:Math ftl (Score:5, Insightful)
If you have a screen that works 90% of the time, and you detain 300 people, 270 will be terrorists.
Congratulations, you got it wrong exactly the way that is being complained about.
The test accuracy is measured compared to the population tested. In fact, a test that consistently says "no cancer" in all cases is 99% accurate when run on the general population.
Re:Simple (Score:3, Insightful)
Re:using a screen that works 90% of the time (Score:2, Insightful)
As you can see, for each potential target, it works 90% of the time. Any other interpretation would be ambiguous.
Article perpetuates the problem (Score:5, Insightful)
The BBC article, while claiming to want to reduce confusion, actually perpetuates the problem by using the meaningless "90%" number instead of the specific positive and negative failure rates. If every article describing tests would quote both failure rates, that would go a long way to getting people to understanding the situation.
Re:Second opinion (Score:5, Insightful)
There's a second opinion in the US when you're put on the no fly list? Or in the UK, when you're detained without charge for weeks (the Government wanted three months)?
The point is that idiots as described in the article think that a "90% scanner" means 90% probability they are guilty, and use to urge action on such people without further checks. And even in a court of law, the point being made is still important: imagine the prosecution telling the jury that the fingerprint/DNA test is 99.99% accurate, therefore he must be guilty? In other words, these further checks are useless if they also fall on the same flawed statistics.
You're okay with your medical analogy, because most doctors have an understanding of basic statisics - unlike the police, politicians, and random members of a jury.
Re:Come On (Score:5, Insightful)
Anyone can write software to look for a turban
This sort of racist bollocks is what has been getting people attacked in the US for wearing them, despite them being an optional part of the Muslim faith so most turban wearers are from entirely different religions which actually require them.
http://en.wikipedia.org/wiki/Turban [wikipedia.org]
Please educate yourself before posting such drivel.
Granted (Score:5, Insightful)
The given version of "terrorist" is arbitrary and thus subject to change over time - from people who hijack planes with guns and explosives, to apparently nowadays, Iceland [nytimes.com], however I think that if you're starting with a number of 1 in 3000 you are so far from reality anyway that what you really want to do is harass innocent people.
Let's look at ALL the hijackings from 1970 to 2000 [bts.gov], a total of 924 hijackings. I couldn't find more recent figures quickly, but let's assume that hijackings have continued at a rate of around 30 per year (the average from 1970-2000), that would add another 30 * 9 = 270 hijackings, for a total of 1194 ok I will be generous 1200 hijackings.
Now let's assume (and this is a BIG assumption - I am again going to be very generous) that TEN people, (the terrorists), board the plane for EACH hijacking event. So now we have 12,000 terrorists.
Now let's just look at the passenger data for the LAST YEAR ALONE for the top 5 airlines [iata.org]. They carried last year 420 million people. LAST YEAR. Now assuming that since 1970 till today there have been a total of 12000 "terrorists" (a VERY generous number), when you divide 420 million by that, you would be looking at 1:35,000 people being a "potential terrorist". However do remember that I am only including passenger data for ONE SINGLE YEAR. Assuming again a 90% accuracy, you are still wrongly intimidating well over 3500 people.
If I was to go through year by year and gouge up the billions of people that have been transported by air, the actual chances of the person being screened actually being a terrorist drops to almost zero.
I will not argue against the value of security as a deterrent. However I think that airport security employees should be well aware that they are, more likely than not, harassing innocent people. Therefore all the excessive bullying, posturing, abuse, privacy and rights violations are completely unnecessary in this context. Airline terrorism is NOT a real threat, be it ever so dramatic on the few times when it does happen. Use technology to screen for the obvious, and lock the god damned cockpit door with a solid lock, for the not so obvious.
screening tests in medicine (Score:3, Insightful)
"The problem is important in any area where a less-than-perfect screen is used to detect a rare event in a population"
Unfortunately, there is no such thing as a perfect screening test for anything in medicine. Some are better than others, but none are perfect. This is a very difficult concept for most people, unfortunately, and for many insurance companies.
It is not such an issue for the better screening tests such as colonoscopy but it is very difficult for things like PSA where there is a large body of evidence it can do more harm than good on average if used routinely even within the recommended ages. For a patient, you're lucky if you can have a meaningful discussion in 5-10 minutes which is an awful large chunk of an office visit that usually has >4 talking points.
It is a problem for doctors and insurance companies because some well intended person with the insurance company will decide to measure the quality of its doctors (which I support in theory) by measuring, for instance, the percentage of age and gender appropriate patients under the care of a given physician that have their PSA checked annually. The problem is, there is absolutely no concensus in medicine that it should be checked regularly as a screening test. I'm not sure I want mine tested when the time comes around unless my family history changes between now and then. So to measure a physician by this marker or other screening tests is fraught with problems, since many patients might opt out for very good reasons. Also, I'm not going to recommend any test because an insurance company wants me to, only if it is right for any given patient.
Bottom line is there are no perfect tests and testing is not always the right thing to do. Most people do not understand that because it is a hard concept to grasp.
Re:The Positive Benchmark Is: +1, Seditious (Score:2, Insightful)
Please stop pissing on Vonnegut.
And yes, pretending to speak for him is pissing on him, regardless of how likely it is that he would agree.
Re:Math ftl (Score:3, Insightful)
Re:Simple (Score:3, Insightful)
Let Circle A be the traveling public.
Let Circle B, intersecting circle A, be terrorists.
Let Circle C, within Circle A, but intersecting Circle B, be the set of those who the test identifies as terrorists.
Any person in Circle C but not in Circle B is a false positive.
Any person in Circle B but not in Circle C is a false negative.
Vary the location and size of Circle C to demonstrate tests of varying accuracy.
This works for terrorists, for cancer, for any test, really. Just wish I could draw it in my post instead of explaining it out.
Re:Simple (Score:3, Insightful)
Let Circle C, within Circle A, but intersecting Circle B, be the set of those who the test identifies as terrorists.
Don't forget: it's possible that B and C don't overlap at all.
Re:Simple (Score:3, Insightful)
Okay, so back to the main article's question - how do you teach them statistics?
I work with a lot of biologists and other people who don't have a clear understanding of probability theory, statistics, etc. and one thing that I've found works very well is to make very clear analogies to simple probabilistic systems that they can understand.
For example, going back to the 90% effective test, imagine that you have a wheel with an arrow on it which is the test. On this wheel, there are 10 boxes, 9 of which say "Not a terrorist" and 1 of which says "You're a terrorist." Now, hand this wheel out to 100s of people and tell them that anyone who lands on "You're a terrorist" gets locked in prison.
A surprisingly large amount of probability theory can be expressed in simple terms like this.
Re:Come On (Score:4, Insightful)
You'd have a really good point if there weren't actually bigoted assholes and/or ignorant people in the world who agree with the great-grand-parent. Earlier in my life I may have been one of them.
I remember painting Muslims with a very broad and unfair brush. People would tell me that all Muslims aren't bad and most want the same thing I do, peace and prosperity. Why don't they speak out against the bigoted extremist representatives then? I would ask.
I didn't have the slightest understanding of the culture and environment those types of ideas breed in and probably still don't. However, I can come out of my own bubble enough to ask myself the question - What motivation would I have to speak out against wrongs being done against a culture who shows repeated disrespect and ignorance for my own?
I'm not suggesting we adopt sharia law and that all North American women start wearing burqa as a sign of respect. There is a very thick line between embracing and adopting a culture and respecting it.
Re:Come On (Score:3, Insightful)
This sort of racist bollocks is what has been getting people attacked in the US for wearing them
What a load of crap, back in 2006 NBC dateline had a bunch of muslims go to a Nascar race and see if they were harassed, guess what they were NOT bothered at all. This sort of idiotic bollocks is what perpetuates the myth that the US is full of racists.
Re:Not possible, at least for now (Score:4, Insightful)
You'll know when your people are ready for statistics. . . don't even bother trying until state-run lotteries go broke for lack of players.
Er, not really. The usual cost-benefit, expected-payoff analysis doesn't really work when you're talking about extreme examples like winning the lottery, at least not with huge payoffs measured in tens or hundreds of millions of dollars. You can know, perfectly well, that the ROI on a lottery ticket is less than the cost of the ticket, and still consider it a perfectly rational investment.
If I buy $150 worth of groceries and throw in a $1 lottery ticket on top of it, the effective cost to me is zero. I'm never going to notice that dollar being gone. Not having that dollar is going to make no difference to my life. But in the (exceedingly unlikely, yes) event that I win a $100 million jackpot, the payoff is damn near infinite. Having that kind of money can't really be compared to, say, getting a raise, or seeing your stocks go up in the market. It's just on a whole different scale.
So in short: infinity - (0 * 10^-9) = infinity. Don't assume that everyone who buys a lottery ticket is ignorant. Actually, I suspect most people who buy lottery tickets are making this kind of calculation, even if they're not doing the numbers quite as explicitly.
Here's an example in the opposite direction, which I think will make things a little more clear. Suppose I were to set up a "reverse lottery," which works as follows. You have, let's say, a net worth of $100,000. If you sign up for my lottery, I pay you a dollar. Then you pick six numbers between 1 and 10, I draw six balls out of urns, and if the numbers match ... I take everything you own. Your house, your car, your computer, the clothes off your back. You're turned out on the street.
In probabilistic terms, it would make perfect sense for you to play. 1 - (100000 * 10^-6) = 0.9, which means that the game has a positive expected payoff. In fact, it would make sense for you to play a lot, up to whatever limit is allowed, let's say once a day. But would you do it? I kind of doubt you would, because every day, you'd be looking at that one-in-a-million chance of having your life shattered. Most people would consider that a bad risk, no matter what the raw numbers say. And people who play the lottery consider it a pretty good risk for the same reason.
Re:Math ftl (Score:4, Insightful)
A broken clock is right at least twice a day.
Re:Simple - the power of celebrity (Score:3, Insightful)
The simplest way to get a message across to the "masses" is simply to have a celebrity deliver it. No explanations, no demonstrations. Simply a script that says: "you know me, I'm that nice, trustworthy person from <name of popular programme> so you know when I speak, I'm telling the truth ..."
People tend to trust individuals they know, they "know" the characters on TV - even though they are actors and probably nothing like that in real-life. It doesn't matter, just think about all the causes that get a celeb on board and then effect political change, even though it's a tiny (but vocal) minority of the population involved and therefore about as non-democratic as it's possible to get.
Re:Come On (Score:4, Insightful)
Yet you failed to learn from that post that making insulting jokes about how anyone wearing a turban in the US can be beaten senseless "because they're a Muslim terrorist" is unacceptable in public
Unacceptable to whom? And why?
As I read the joke, it is making fun of ignorant American rednecks. While I guess some people might find painting such people with such a broad brush insulting or offensive, I don't see it, myself.
Apparently you a) have a different interpretation of the joke and b) feel that your interpretation justifies declaring the joke "unacceptable in public." I don't get your logic, and I certainly don't appreciate your arbitrary and unjustified declaration regarding what is or is not acceptable behaviour "in public".
This is particularly true since /. has a significant world-wide readership--if you clowns can't control your bigots that's your problem, not justification for declaring, in typically American imperialist fashion, what is and is not acceptable here in this international, albeit US-dominated, forum.
Ok, the problem with this comment is that it is now a) exactly what I feel and b) -1 flamebait. Oh well.
Should you be tolerant to the KKK too ? (Score:1, Insightful)
After all, "not all KKK members are terrorists" is certainly a truth.
Please tell me, should one be tolerant against people who have an intolerant ideology, as indicated by said ideology prohibiting inter-group relations and promoting attacks against outsiders, like the KKK does ... or islam does ?
Anyone who believes
O you who believe, do not take Jews and Christians as friends; these are friends of one another alone. Those among you who ally themselves with these belong with them. GOD does not guide the transgressors.
(quran 5:51) is not a tolerant individual. Any muslim who treats any non-muslim in a friendly manner commits treason. In a number of nations this will get you the death penalty, especially for women, due to islam specifying it.
Since this is part of the most essential part of islam, anyone who does NOT believe this is not a muslim.
Ergo one is either religiously intolerant Xor one is not a muslim.
Being a tolerant muslim is like being an innocent murderer, a vegan game barbecue : a contradiction by itself.
Should you be tolerant against people who believe this is the way to live ?
Of course, since racism and religious intolerance is a core part of islam itself, that is the exact same question as, should you tolerate muslim ideology ?
So, please tell me, which of these is true :
1) we do not allow for religious intolerance, and therefore we do not allow muslims and/or islam
2) we allow for intolerance, in a non-racist way, and we show the same respect to KKK members as to muslims
3) we allow intolerance in a racist way : some groups get to be racist, others get punished
Obviously 3, your choice, is the worst of all in the effect it'll have on racism. It both makes you yourself a racist and encourages racism in others.
Needless to say, we have apparently chosen the dumbest of these options, number 3. Of course, before 1941 there was no politician in public office who dared call Hitler anything but the "champion of the poor".
The ideology of islam is as much an enemy of human rights, of equality and of tolerance as nazism. Please treat it as such.
Sally Clark (Score:4, Insightful)
I think that's called the "prosecutor's fallacy." If there's a 1/10,000 chance of a child dying of cot death, and a woman has two children die of cot death, the prosecutor tells the jury that the chances could only be 1/10,000 * 1/10,000 = 1/100 million that both deaths were a cot death, so she must have murdered them.
This only works if the deaths are statistically independent, which they're not. The parents could have a genetic defect which cause 2 successive infants to die.
If each parent had 1 fatal recessive genetic defect, then 1/4 of their children would die, so the odds are 1/16 that two successive children would die. But actually a lot of fatal birth defects are more complicated than that simple mendelian pattern.
It's even more complicated because some mothers have been captured on video trying to smother their children.
Conditional probability... (Score:1, Insightful)
The probably easiest way would be to somehow force people to only report the probability conditional on the fact they are tested positively, i.e. compare P(Not being a terrorist | identified as terrorist) to P(Being a terrorist | identified as terrorist)
e.g. Instead of saying a test has 90% chance to be positive (as stated), report that the test is wrong 90% of the time.
Additionally, I wonder if there ever was any real terrorist that answered truthfully to the question "Are you a terrorist?" which they ask you when you fly to the US...
Also a problem for car efficiency, other ratings (Score:4, Insightful)
Take an example where a Range Rover gets 14 MPG, a Toyota Rav4 gets 24 mpg, and a Prius gets 46 mpg. It isn't intuitive based on the miles per gallon, but moving from the Range Rover to the Rav4 saves more fuel than moving from the Rav4 to the Prius. That is because people don't drive a fixed number of gallons, but drive (more or less) a fixed number of miles. When you look at the gallons used per 100 miles it is clear. The Range Rover uses 7.14 gallons per 100 miles, while the Rav4 uses 4.17 and the Prius 2.17. So it is clear that changing from a Range Rover to a Rav4 will save almost 3 gallons per 100 miles, while changing from a Rav4 to a Prius only saves 2 gallons per 100 miles.
You Need the Full Confusion Matrix... (Score:4, Insightful)
... and a utility function too!
The article is confusing because it doesn't indicate the false negative rate. You basically need to know the entire confusion matrix before inferring anything. This way, you can not only calculate the accuracy and the false positive rate, but you can also calculate the false negative rate, the precision and the recall. Precision and recall are much more useful metrics than recall when it comes to tests like these.
Also, you need to know how much it really costs you to have false negatives and false positives. If you accuse someone erroneously of being a terrorist, and the only inconvenience is a few extra minutes of body search (and the humiliation) at the airport, it *might* still be worth the trouble. If on the other hand you end up sending the poor dude to jail, and he sues you for wrongful conviction, then not so much. You therefore need to have a utility function that assesses the cost of getting it right and wrong both ways (positive and negative). That's basically what is discussed in the other article (the cost of cancer screening tests), albeit in an informal way.