typodupeerror

## Why Improbable Things Really Aren't166

First time accepted submitter sixoh1 writes "Scientific American has an excellent summary of a new book 'The Improbabilty Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day' by David J. Hand. The summary offers a quick way to relate statistical math (something that's really hard to intuit) to our daily experiences with unlikely events. The simple equations here make it easier to understand that improbable things really are not so improbable, which Hand call the 'Improbability Principle:' 'How can a huge number of opportunities occur without people realizing they are there? The law of combinations, a related strand of the Improbability Principle, points the way. It says: the number of combinations of interacting elements increases exponentially with the number of elements. The 'birthday problem' is a well-known example. Now if only we could harness this to make an infinite improbability drive!"
This discussion has been archived. No new comments can be posted.

## Why Improbable Things Really Aren't

Mr. Hand

Mr Hands?

• #### Improbability drive (Score:4, Funny)

on Tuesday February 18, 2014 @04:11AM (#46274043)

How improbable is the Heart of Gold?

And Zaphod stealing it...

• #### Re: (Score:3)

Obviously Zaphod successfully stealing such a thing was phenomenally improbable... and thus became inevitable as soon as they revved up the first drive prototype.

• #### Re: The day before Fukashima happened (Score:5, Interesting)

by Anonymous Coward on Tuesday February 18, 2014 @04:19AM (#46274055)

The day before Fukashima happened I was writing a paper for an Industrial Safety class on the subject of Nuclear safety. My conclusion essentially made the argument that "Although individual improbable events are unlikely, the shear number of opportunities to experience an improbable event on a day to day basis are staggering." Any specific improbable event is highly unlikely to occur, but the occurrence of improbable events in general is a practical certainty.

The next day I saw on the news that mother nature had done her best to prove my point. The timing worked out to be an incredibly unlikely coincidence, but on a daily basis I rarely notice when unlikely coincidences fail to occur. :)

• #### Re: The day before Fukashima happened (Score:5, Insightful)

on Tuesday February 18, 2014 @05:20AM (#46274193)

There are well defined techniques for measuring the probability of events happening in industrial safety. Safety Integrity Levels or SIL are used to categorize the possibility of a life threatening event occurring.

The problem is how low a risk do you need and how much will it cost you to get there. Fukashima would probably not have happened if the sea wall had been higher, but the designers had to make the judgement that it was not worth the millions of cost required to build a bigger wall compared to risk of it being breached. Unfortunately decisions like that in hindsight always look flawed.,

• #### Re: The day before Fukashima happened (Score:5, Funny)

by Anonymous Coward on Tuesday February 18, 2014 @05:57AM (#46274265)

They didn't need higher sea wall. A wall can permanently hold water away and that is overdoing. What they needed to do was to make a watertight, anchored to the ground building for auxiliary generators and connect them to main building by undersea power cables. Basically, build a submarine on land, complete with snorkel. When there is a water surge, it holds generators safe and dry so that they function after the water recedes.

• #### Re: (Score:2)

Any type of working aux generator would have protected it. Although I think a better design would be to have one a few miles inland. That would provide better protection from other natural disasters, accidents, or intentional attacks.
• #### Re: (Score:2)

When I first got into safety engineering I always imagined myself explaining the concept of a SIL level or the ALARP principle to a grieving widow. On an academic level I know exactly why it makes sense and that it's the best thing overall but it took a while to get rid of that feeling in my gut that it wasn't right.

• #### Re: (Score:2)

Are there any well defined techniques for determining how may possibly fatal low probability events are extant? I doubt it. You may be able to say "Of the things we've considered, there are these things which each have this probability", but you can't calculate the things you haven't considered...which is most of the universe. Granted the liklihood of being stomped on by Godzilla is too low to consider, there are lots of things "metaphorically similar to Godzilla" that are more probable.

(I'm saying Godzi

• #### Re: (Score:3, Insightful)

by Anonymous Coward

Just a bit of a nitpick. Mother nature did not "prove your point". Statistics infer data for a population from a sample. A single event from that sample does not prove or disprove anything about the population, nor the sample. Had there not been an event at Fukushima that day, your statement would not have been any more true or false, or any less proven. Your point is proven with statistical significance tests on the sample, not by taking one event and saying "here's proof". That's the opposite of statistic

• #### The probabilities of multiple cot deaths (Score:5, Interesting)

<ph1ll1phenry AT yahoo DOT com> on Tuesday February 18, 2014 @08:00AM (#46274495)

Another example is in the curious case of Professor Meadows [wikipedia.org] - a great paediatrician but a shite mathematician.

He endorsed the dictum that “one sudden infant death is a tragedy, two is suspicious and three is murder, until proved otherwise“. The trouble is, given enough numbers, multiple cot deaths are an inevitability.

Unfortunately, his expert testimony convicted an innocent woman. Fortunately, she was released on appeal when the math was reviewed.

• #### Re: (Score:2)

One of the issues is the conflation of time with probability. A coin flip is half odds whether you flip it every second or every ten thousand years.
However, when you flip a coin every second for ten thousand years,you get different results. The million monkeys typing a million years to produce Shakespeare is a perfect example of how multiplying probability by time is like dividing integers by zero. Things get funky.

If the Fukushima risk analysis looks at one event per day versus one event per hour or on
• #### Re: (Score:2)

Actually, the million monkeys typing a million years is a point against things getting funky with time. Say Hamlet is 100,000 characters long, say the typewriters only consist of 52 characters (lower and uppercase, let's forget about punctuation). This means that a monkey typing a random 100,000 characters has a probability of 1 over 52 to the power of 100,000 to produce Hamlet. The monkey can bang away for a million years (10^6), he can invite a billion friends (10^9*10^6=10^15), they can bang away for a f
• #### Re: (Score:2)

What would really be funny is if two days later you got an F on the paper because the teacher didn't feel enough evidence existed to prove your point.
• #### 42 (Score:5, Insightful)

<`slashdot2' `at' `anthonymclin.com'> on Tuesday February 18, 2014 @04:31AM (#46274079) Homepage

My theory of the question for life, the universe, and everything.

The books rely heavily on probability (even as far as powering the faster than light engine as alluded in the summary).

A pair of dice is one of, of not the most common symbol for probability, chance, and luck (at least in Anglo-American culture). And how many pips are on a pair of dice?

• #### Re: (Score:2)

Nice catch! That's *got* to have been intentional. It's way too improbable... to be... coincidence...

Damn. I think there's a flaw in my illogic somewhere.

• #### Seems legit (Score:5, Funny)

on Tuesday February 18, 2014 @04:45AM (#46274107) Homepage

I found it highly improbable that an article on that topic could be boring. It explained to me in laborious detail why I was wrong.

• #### A million to one (Score:2)

Did someone else notice that if the chances for something to happen are exactly a million to one, there is a 1 to ten chance that it actually happens?

• #### Re: (Score:2)

many people in my line of work have decided to use a DateTime value as a primary key in a database. i always see the jr programmers doing this. i'm guilty of it myself. you think, "it's impossible that two users will create records at the exact same nanosecond." You quickly learn how probable improbable things are.

just explaining it to the jr programmers never seems to be enough. they never really appreciate it until they actually screw something up.
• #### Re: (Score:2)

Did someone else notice that if the chances for something to happen are exactly a million to one, there is a 1 to ten chance that it actually happens?

Million-to-one chances work nine times out of ten.

• #### Re: A million to one (Score:2)

I stand corrected.

• #### Summary. (Score:4, Insightful)

on Tuesday February 18, 2014 @05:15AM (#46274175)

Why? Because there are 7 billion people on Earth.

• #### This is why TV news is toxic (Score:2)

At any given time there are floods, fires, murders and any possible crime happening somewhere in the world. People, however are designed to react to what's happening in their community -- in their immediate environment.
Having every horror happening, nationwide, shoved down your throat 24X7 is equivalent to poisoning yourself.
• #### Lottery scratch tickets; not so random (Score:5, Interesting)

on Tuesday February 18, 2014 @05:41AM (#46274243) Homepage Journal

Very interesting article on it http://www.lotterypost.com/new... [lotterypost.com] been a long time since I've read it (bookmark), but this guy can tell which scratch tickets will pay off by by reading their serial numbers, winning wasn't as improbable as one is led to believe - and yes, of course he's a statistician.

I don't play the lottery, maybe a ticket twice a year, but my son likes the scratch tickets, I told him that they were predictable, he refused to listen; he wouldn't even pick up the link I printed out. He refused to imagine that it wasn't anything but random. It was just an odd reaction, I can't begin to explain the reasoning behind it.

The link is old so I imagine the serial number gig has been fixed (yet I have no clue one way or the other), but supports the improbability disclaimer.

• #### Re:Lottery scratch tickets; not so random (Score:4, Interesting)

on Tuesday February 18, 2014 @09:58AM (#46274987)

The link is old so I imagine the serial number gig has been fixed (yet I have no clue one way or the other), but supports the improbability disclaimer.

While this may be interesting to some, it has very little to do with TFA.

TFA is arguing that random events are often more probable than we might think, because we often fail to take the context of an event into account.

Most of the scenarios in TFA are variations on the "birthday paradox," which basically amounts to people looking at an event X with a very tiny probability P in a specific case, and assuming that P is the probability it would happen. But we often forget that there are Q number of combinations or situations that would all result in X being true... so P is a gross underesimate of the probability of X.

Your link deals with a poorly designed computer algorithm that actually isn't random which is spitting out lottery tickets. The scratch-ticket system has to make money, so the numbers can't be entirely random -- they must only payout so many tickets within a given batch. The guys who designed the computer system that chooses the numbers didn't take into account that there were statistical clues that could allow someone to "crack the code" to the fake randomness.

There are two completely different phenomena. Finding a flaw in pseudo-randomness is completely different from miscalculating odds of genuinely random events.

• #### Re: (Score:2)

The link is old so I imagine the serial number gig has been fixed (yet I have no clue one way or the other), but supports the improbability disclaimer.

While this may be interesting to some, it has very little to do with TFA.

TFA is arguing that random events are often more probable than we might think, because we often fail to take the context of an event into account.

Most of the scenarios in TFA are variations on the "birthday paradox," which basically amounts to people looking at an event X with a very tiny probability P in a specific case, and assuming that P is the probability it would happen. But we often forget that there are Q number of combinations or situations that would all result in X being true... so P is a gross underesimate of the probability of X.

Your link deals with a poorly designed computer algorithm that actually isn't random which is spitting out lottery tickets. The scratch-ticket system has to make money, so the numbers can't be entirely random -- they must only payout so many tickets within a given batch. The guys who designed the computer system that chooses the numbers didn't take into account that there were statistical clues that could allow someone to "crack the code" to the fake randomness.

There are two completely different phenomena. Finding a flaw in pseudo-randomness is completely different from miscalculating odds of genuinely random events.

Accepted.

• #### Re: (Score:2)

As far as the old article is concerned, the problem was that the Lottery commission, in order to maintain sales, interfered with the actual randomization. Every pack of 1,000 tickets sent to a store has so many \$2, \$5, and \$20 winners. A clerk at the store paying attention would open a new pack of a thousand tickets and keep track of the winners. If there were fewer than expected, then it actually made sense to buy the last 150 tickets of the pack (using friends and accomplices) This is best done at the sor
• #### Re: (Score:2)

As far as the old article is concerned, the problem was that the Lottery commission, in order to maintain sales, interfered with the actual randomization. Every pack of 1,000 tickets sent to a store has so many \$2, \$5, and \$20 winners. A clerk at the store paying attention would open a new pack of a thousand tickets and keep track of the winners. If there were fewer than expected, then it actually made sense to buy the last 150 tickets of the pack (using friends and accomplices)

Nope. While this may be another way to hack some lottery tickets, this is not what happened in the GP's link scenario. You can read more of the details about the statistician who publicized the problem here [wired.com].

Basically, on the tickets in question, there were a lot of exposed numbers on the tickets that were visible before you bought the ticket. There were a few hidden numbers that you scratched off after you bought the ticket and tried to match to the ones that were already visible.

The problem was that

• #### Re: (Score:2)

but this guy can tell which scratch tickets will pay off by by reading their serial numbers, winning wasn't as improbable as one is led to believe

CSB:
My elementary school set up a sort of lottery during a yearly festival.

So two classes were tasked with preparing the winning and losing lots for the lottery by writing the kind of price or something like "no win" on little paper squares.
They then folded the paper squares and stapled them shut so you couldn't tell what was written inside.
During the festivities the kids ran a stand were you could buy and draw the lots from a couple of big, open bowls.

Almost all of the prices went to a handful o

• #### Re: (Score:2)

but this guy can tell which scratch tickets will pay off by by reading their serial numbers, winning wasn't as improbable as one is led to believe

CSB:

Well, the winning and losing lots had been prepared separately and, not thinking about it and lacking the direct comparison, the teachers in charge of the two groups had been unaware that their staplers were loaded with silver- and copper-colored staples respectively.

So by looking at the color of the staples, you were able to pick only winners out of the open bowls.

That required one to look at the paper squares, I can go one better.

A new computer store had a drawing for an Osborn, I put in a slip in hopes of a win (duh).

I had high hopes so showed up for the drawing. A family was there just looking; they asked the teen daughter if she would pick out a slip. Surprised and fairly embarrassed she reached in and pulled out a wad of paper and the winner.

Since that day I've crumple up any drawing entry to make it larger and more accessible.

At a elementary school event they h

• #### Re: (Score:2)

Yeah, well, we run raffle/lotteries several times a month. The wadded-up mumbles of paper are *not* the ones that get pulled out when we're fishing for winners. Just sayin'.

Nor are the illegible scribbles even considered as possible winners. "Penmanship, people! PENMANSHIP!" If I can't read your winning name, then you ARE NOT the winner.

This is especially funny because our market is elementary school teachers. Yes: the ones who should be teaching children how to write legibly.

Yeah, right...

• #### Roll a dice... (Score:2)

Roll a dice. Each of the outcomes only has a probability of 16.p6 % (assuming a fair d6), which is fairly unlikely. Yet, there's a 100 % probability that you will obtain one of these unlikely results.

• #### Re: (Score:3)

Roll a dice

Die. Dice is plural. Dice.com sucks. Die, Dice, Die! Wait, what were we talking about?

• #### Re: (Score:2)

As far as I can remember: the 'correct' singular of 'dice' is 'dix'!

• #### Re: (Score:2)

Roll a dice. Each of the outcomes only has a probability of 16.p6 % (assuming a fair d6), which is fairly unlikely. Yet, there's a 100 % probability that you will obtain one of these unlikely results.

unless one die rolls into a crack or becomes a leaner against the table or rolls under the refrigerator in which case there is a result but it's more along the lines of Schrodinger's cat or Christ could come back to earth and take up your friends in rapture along with the dice and leave you hanging or the universe could explode. I'd give it five 9s instead. 99.999%

• #### Oblig XKCD (Score:3, Interesting)

by Anonymous Coward on Tuesday February 18, 2014 @06:06AM (#46274279)
• #### Re: (Score:2)

Same information, but the visual aspect of the animated GIF is somehow much more accessible. One more data point on how the human brain is so poorly adapted to statistical inference as compared to our natural abilities with visual information like "is that tiger going to eat me", or "can I make it across the gap between this tree and that tree when I jump".

• #### People round down (Score:4, Insightful)

on Tuesday February 18, 2014 @07:35AM (#46274431) Homepage

Often when the probability of an event gets close to 1-in-100 people just say "impossible", i.e. they round down to zero.

They also forget that one can increase the chances of the event happening by repeating the trial. E.g. funding a 1-in-100 chances of blow-out-success company sounds like a risky bet, but if you fund 100 such companies, it is a rather safe bet. Hence VCs.

This is a counter-intuitive situation in which increasing the occurrences of the risky behaviour makes the whole situation safer. (Contrast this with Russian roulette in which increased trials is definitely a bad thing).

• #### Re: (Score:2)

The way I like to summarize it when talking to non-technical types is this: The odds of any one ticket winning the lottery jackpot are astronomically small. Regardless, people win the jackpot quite regularly.

Low probability per trial × many trials = reasonable probability of occurrence overall.

Rounding small probabilities down doesn't fully explain all the ways folks get tripped up thinking about probabilities. For example, the Birthday Paradox doesn't fit that model directly, because it's counter-

• #### Re: (Score:2)

imo,l the difference between statistics and probability is that every customer in a casino is gambling, but the casino itself is most definitely not gambling. It is the difference between owning one hand of cards and owning ten thousand hands.
• #### I still want to know how to ... (Score:2)

generate a small amounts of finite improbability .... to break the ice at parties by making all the molecules in the hostess's undergarments leap simultaneously one foot to the left, in accordance with the Theory of Indeterminacy.

• #### As the Venerable Terry Pratchett says: (Score:2)

Everybody knows that that vital million to one chance happens nine times out of ten.
• #### This article would have been more useful if it (Score:5, Insightful)

on Tuesday February 18, 2014 @08:40AM (#46274589)

applied to debunking so-called "Intelligent Design". There are a few high profile proponents who claim that the probability of an organism as complex as humans evolving from single celled ancestors is so small as to be impossible, therefore we must have been "designed" by "someone" (a variation on the God of the gaps principle used by others for the same purpose). They like to point out eyes as organs that are so complex they could not have evolved, even though we have numerous living organisms that have organisms with photosensitive sensitive organs that aren't quite eyes, perhaps on their way to becoming eyes, many generations/mutations down the road.

In a single field of view under a microscope I can see tens of thousands of bacteria swimming around in a drop of water. Multiple that by all the drops of water in the world and you quickly realize that the number of living organisms is a HUGE number. With all that genetic replication (with errors that sometimes result) and gene swapping going on, and all the DNA floating around freely in the waters of the world, it seems inevitable that there will be enough mutations taking place to produce the variety of life we see on earth.

• #### Re: (Score:3)

Along the same lines, I remember first encountering this concept of probability during the "Bible Code" craze many years ago. The credulous gushed that the odds of this or that secret message being in the Bible by chance are some billions to one against, and the rebuttal was that there were many billions of places in the Bible to look for it, so the odds were actually pretty good that you would find it somewhere. I think ultimately someone made a page demonstrating how to find Shakespeare using the same tec
• #### Re: (Score:2)

I think ultimately someone made a page demonstrating how to find Shakespeare using the same technique. Or maybe finding the same messages in the works of Shakespeare.

Well, the classic debunking occurred by finding similar patterns in Moby Dick, but there were also things found in War and Peace (particularly the Hebrew translation!), as you can read about here [anu.edu.au].

This has always been my answer to the "Strong Anthropic Principle" which claims that some agency must have tuned the universe to be able to support conscious life. Since no one knows how many repetitions exist, the SAP has no legs to stand on.

While I think SAP is a bogus argument, your rebuttal also makes little sense.

(1) It doesn't matter "how many repetitions exist" -- it only matters what the odds are, and whether that seems a reasonable way to evaluate the nature of the universe. I can draw a 1 in 1,000,000 poker hand the first time I ever play c

• #### Re: (Score:2)

It doesn't matter "how many repetitions exist" -- it only matters what the odds are

Incorrect, this was the point of the article. If the odds are one in a billion, but you have a billion repetitions of the event, the odds of a hit are much higher than one in a billion. I agree there is little reason to lend credence to any given estimate of how likely or unlikely a human-friendly universe is. My point was that, even if you had a good measure of the odds, saying "the odds of a human friendly universe are a trillion to one against" is not illuminating since you do not know how many universes

• #### Re: (Score:2)

Incorrect, this was the point of the article. If the odds are one in a billion, but you have a billion repetitions of the event, the odds of a hit are much higher than one in a billion.

Actually, no -- that was NOT the point of the article. The article mentions the law of large numbers at the beginning, but the rest of the article is not simply about many chances at an event = higher probability.

Instead, the article is about variations on birthday paradox, that is, if I attend a small party and find that someone has the same birthday as I do, everyone at the party is often surprised. Your response is that if I attend hundreds of parties or if there are hundreds of guests at the party,

• #### Re: (Score:2)

One last thing: I know that speculative physics (particularly string theory) is actively involved in trying to imagine stuff outside of our universe -- other dimensions, brane theory, etc. Some of these models have been created to try to explain problems in the way physics seems to work in our universe.

Once we have some actual emprical evidence of some sort of interaction from outside the universe that actually gives some merit to these speculative mathematical theories, I'll be the first one to step up

• #### Re: (Score:2)

It seems we disagree about what constitutes a combination. I would consider dealing a new hand of cards, for example, to be just another combination. The birthday problem is only a surprise to some because they do not realize the number of combinations, i.e. the number of discrete 1/365 events that are repeated. Regardless, I never said anything about having any evidence of multiple universes, I said "do not know".
• #### Re: (Score:2)

Thinking about it, your refutation is probably better. An improbable event happened, this doesn't prove anything related to design.
• #### Re: (Score:2)

Ah, sorry. I blathered on a bit there, assuming the birthday problem idea was unclear. I think TFA was trying to make a distinction between actual improbable events (which only tend to happen with loads of trials) vs. apparently "improbable" events which actually happen frequently because people don't know how to calculate probability. It was mostly about the latter. I assumed from your wording that you were talking about the former. In any case, I understand what you were saying now.
• #### Re: (Score:2)

Which makes me think of Stairway to Heaven; I checked, the Jesus freaks were right. But there's no way possible that anyone could have done that on purpose. It's clearly just a wild coincidence.

I remind the Jesus freaks that a prayer said backwards is a prayer to Satan, so that song is in fact a Christian song.

Ever hear Helter Skelter backwards? It changes to "I like smack" (and, well, that one may have been deliberate).

• #### Re: (Score:2)

it seems inevitable that there will be enough mutations taking place to produce the variety of life we see on earth.

Hold on a sec. Just because we might significantly underestimate the probability of something is NOT evidence that it is "inevitable" (or even highly likely).

Note that I'm on the side of evolution here. And I think the "intelligent design" movement is largely a smokescreen to get religious teachings back into schools.

But we still have to be careful about skewing our perceptions of odds the opposite way. If we don't EVER accept the possibility of design, then we must assume that a pocketwatch found bu

• #### Re: (Score:2)

The arguments that the proponents of 'Intelligent Design' use to attempt to discredit Evolution, are far more effective at showing the absurdity of a 'Creator'!

• #### Been trying to make this (Score:2)

This is a point I bring up occasionally in regards to the so called "war on terror". The thing is these highly rare events, on average, don't happen. Your chances of ever encountering an attack is nearly nil. However, given long enough time spans, and large enough areas, they do happen with occasional frequency.

That is the thing, you can expect anything that could happen is going to happen occasionally given a large enough population that it could happen in and a long enough time for it to happen.

So if you

• #### winning the lottery twice (Score:2)

The fact that people win big lotteries twice in a lifetime (sans any fraud) still blows my mind. If that can happen, just about anything can.

• #### There already is a word for this.... (Score:2)

It's called 'probability'. Yes people don't understand it well, but inventing new terminology isn't the answer.

• #### It was a million-to-one shot (Score:2)

But everyone knows that million-to-one shots occur nine times out of ten.

• #### You didn't need to go throught all this trouble . (Score:2)

Dr. Manhattan told the world in 1987, "Only what can happen does happen".
• #### Simple. I'll make some Tea. (Score:2)

Now if only we could harness this to make an infinite improbability drive!

From HHGTTG, quoting from here: Infinite Improbability Drive [wikia.com]:

The principle of generating small amounts of finite improbability by simply hooking the logic circuits of a Bambleweeny 57 Sub-Meson Brain to an atomic vector plotter suspended in a strong Brownian Motion producer (say a nice hot cup of tea) were of course well understood

If ... such a [infinite improbability] machine is a virtual impossibility, it must have finite improbability. So all [one has] to do in order to make one is to work out how exactly improbable it is, feed that figure into the finite improbability generator, give it a fresh cup of really hot tea... and turn it on!

• #### Remember (Score:2)

" magicians have calculated that million-to-one chances crop up nine times out of ten."

Terry Pratchett

#### Related LinksTop of the: day, week, month.

Adding features does not necessarily increase functionality -- it just makes the manuals thicker.

Working...