Fooled by Randomness 291
| Fooled by Randomness | |
| author | Nassim Nicholas Taleb |
| pages | 220 |
| publisher | Texere |
| rating | 8 |
| reviewer | Max Tardiveau |
| ISBN | 1587990717 |
| summary | Debunking fallacies of observation, Taleb reminds us of the pervasive ineffables that complicate life at every turn. |
The main topic of the book is the fact that all humans are simply terrible at judging probabilities. Taleb is a securities trader, so a lot of the book revolves around financial probabilities, but his argument is mainstream and requires absolutely no knowledge of the markets. The book details examples of people wildly misjudging risks and probabilities in many contexts. Often that misestimation is understandable in advance of certain events, but harder to excuse after they've occurred; Taleb hits pretty hard on what he calls "data snoopers," his term for people who back-fit theories to existing data.
One of the most notorious examples is the Bible code (which has been thoroughly debunked), but Taleb argues that analysts who spend their time trying to find patterns in historical market data are no different: if you try long enough and hard enough, you will unavoidably find apparent regularities, which can be extremely compelling when seen in isolation. In context, though, they dissolve into nothing but meaningless statistical anomalies. Taleb rightfully compares these searches for meaning to the famous monkeys and typewriters parable.
Taleb's best example of poor probability intuition is probably the infamous survivor bias, which is our tendency to be disproportionately impressed by success. We almost always ignore the fact that, for one success story, there are many failures. But we seldom hear about the failures (just like we never hear about the many theories that didn't fit the data). So it's all a game of numbers: out of 10,000 traders, a few are statistically bound to be successful, even if they are nothing more than lucky idiots. The fact that they succeeded does not mean anything. It doesn't mean that they are bad traders, but it doesn't mean that they are good traders either, because on average somebody had to succeed.
One of Taleb's hot buttons is that people tend to be too confident in what they know. He argues convincingly that we should take everything, including science, with a grain of salt. Writing about Karl Popper, he points out that there are only two kinds of scientific theories: those that are demonstrably false, and those that are not yet demonstrably false. An irksome (but sadly true) observation, yet most people behave as if what they know is eternal truth. One could of course argue that Popper's observation is but another kind of truth, but I tend to put a lot more trust in people who question what they know than in people who don't.
Another of Taleb's peeves is the human tendency to over-attribute every random event (the old post hoc, ergo propter hoc). For instance, a commentator saying that "the Dow went down ten points today on concerns about Iraq" is talking nonsense: there is no way anyone can tie such a small market move to any particular reason. I found this specific point (which in retrospect is blindingly obvious) especially enlightening, as I am embarrassed to admit that, until now, I just accepted those market comments at face value.
Taleb also has some fun at the expense of economists and analysts, especially those whose predictions turned out wrong, but who claim that the theories were in fact right, and that the facts simply weren't supposed to be that way. This is what he calls denial of history, and is common among investors and gamblers (the two being of course close cousins).
The style of the book is informal and funny, and often meandering. We hop from one topic to the next, which occasionally may detract from the book's continuity, but overall the author's points come through loud and clear. Ironically for a man who advocates self-doubt, Taleb is starkly self-confident, though not in an irritating way.
Taleb is an intriguing, multi-cultural, iconoclastic character that has been around Wall Street for a while, and now runs his own small firm. Malcolm Gladwell (author of The Tipping Point, an absolute must-read for anyone who owns a brain) has written an excellent article that shows how Taleb's reasoning runs counter to just about every bit of conventional Wall Street wisdom. If you're interested in the markets, especially derivatives, and how Taleb trounces most of Wall Street's voodoo doctors, this moderately technical interview from 1996 is worth reading too.
Overall, a warmly recommended book.
You can purchase Fooled by Randomness from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Dumb Luck (Score:3, Offtopic)
There are times where I feel like I should have died but I haven't. Falling off a ravine into a river, losing control of my car in a turn with traffic on the other side. Just dumb luck.
Fooled by Randomness (Score:5, Funny)
Good point! (Score:4, Funny)
Good point. I'd say I'm that good and
(ughhhh)
NO CARRIER
Re:Good point! (Score:3, Funny)
Script [uibk.ac.at]
M@
Conservative/Liberal take on it (Score:5, Interesting)
Thats called the Just World mentality. Its not a Just World
I've found this is the single biggest commenality in partiasian schools of thought:
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
Liberals go the other way. Luck, circumstance, and opportunity play huge roles in where you are. Poor people are poor because of luck and circumstance. People get hit by trains because they might have just plain been unlucky. Your situation is a result of your environment, including dumb luck.
Personally, this is the single biggest reason I can't stand conservatives. It bothers me to no end how capable they are of assuming that anybody in a bad situation is there because they deserve it.
Yep (Score:2, Interesting)
Personally, this is the single biggest reason I can't stand liberals. It bothers me to no end how capable they are of assuming that anybody in a bad situation is there because of bad luck or circumstances beyond their control.
Re:Yep (Score:2)
Re:Conservative/Liberal take on it (Score:5, Funny)
Re:Conservative/Liberal take on it (Score:5, Insightful)
Re:Conservative/Liberal take on it (Score:2)
Re:Conservative/Liberal take on it (Score:5, Insightful)
The truth is that while some people may become successful or suffer terribly largely through luck, there are situations where you can drastically improve or destroy your life through your own choices.
You can't blame everything on circumstance, but you can't take credit for every good thing that happens to you, either.
That's why I can't stand cardboard conservatives or cardboard liberals. The world is so much more complicated than either of them give it credit for.
(Yeah, I know, off-topic
Re:Conservative/Liberal take on it (Score:3, Interesting)
Huh ? this seems to be a perfect plot for a movie. Wait - nevermind
Choices. (Score:3, Interesting)
It's why people from "better" backgrounds can recover from poverty quickly - why good business-people can create new businesses from catastrophes repeatedly - and why very hard working people who don't know how to work tactically can still die poor. Knowledge and social networks are what keeps one out of poverty, not hard work.
As an aside, drug addiction exists in that gray area between choice- and not-choice, since it can be thought of practically as a "disease of the choice-making mechanisms of the mind." When drug addiction is responsible for poverty, there's little chance of escaping the poverty without addressing the addiction; likewise for mental illness.
Re:Choices. (Score:3, Interesting)
It could be equally argued that they were in those "better" backgrounds because of some genetic advantage. You'd really need to do a large number of controlled studies to determine the order of the cause and effect there. My suspicion is that both genetic propensities and social frameworks play a part.
Knowledge and social networks are what keeps one out of poverty, not hard work.
Hard work is an important element, and you can't just discount it out of hand.
As an aside, drug addiction exists in that gray area between choice- and not-choice
What is a choice, then? I know smart people who are "addicted" to EverQuest to the point that their work quality seriously suffers. Are they making a choice? If I go home early to watch TV or read a book, am I making a choice or just a slave to some electrochemical impulse in my brain?
WHOA, dont let this spiral out of control (Score:2, Insightful)
They are just generalizations, and it's too bad that I'm gunna get modded down to hell for it.
I probably should have left off that last line. It's still a valid point, for the most part.
Comment removed (Score:4, Insightful)
Mischaracterization of conservativism (Score:5, Informative)
This viewpoint is absolutely not essential to conservativism. It is unfair to characterize conservativism as people who punish the poor because they "deserve it."
There is a legitimate conservative idea that many well-meaning social welfare programs actually trap people in poverty by encouraging failure and discouraging success. Clinton famously "ended welfare as we know it" because he believed this kind of reform was the right thing to do.
Of course there is also a wrong kind of conservativism (practiced by many Libertarians and Republicans), in which welfare is cut, while job training programs are cut as well.
Centrists of both parties (such as Clinton) recognize and co-opt the good parts of conservativism and of liberalism. When you say that you can't stand conservatives, I think what you really mean is that you can't stand the self-righteous, self-promoting bigots who make a mockery of the ideology they unfortunately claim to represent.
Re:Conservative/Liberal take on it (Score:5, Insightful)
I've found this is the single biggest commenality in partiasian schools of thought:
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
As a conservative, I'll take the bait. I think that "deserve" is too complex an issue to really know (abused as a child, ends up as a criminal, etc.). It is a core conservative belief, however, that people's fates are in their own hands. I certainly won't sign on to that smart and hard working people automatically get rich, but I've had pretty extensive personal experience with why people end up poor:
#1 reason: being part of a couple that gets pregnant outside of marriage. If you're the woman, you now have the financial burden of a child without the full support of a father. Additionally, you don't enjoy the economies of scale enjoyed when two or more income earners live together. If you're the father, the latter point still applies, and you have child support to pay.
#2 Reason: not completing high school. In our society, for better or worse, this is a signal to all employers that you haven't got your shit together. There are, however, many cases where high school dropout succeed as entrepreneurs.
#3 Reason: Drug / alcohol abuse. No explanation necessary, presumably.
Here's the defining characterstic of a conservative: the belief that if someone works hard and consistently, they can improve their condition. Not everyone who believes this is a conservative, but if you don't believe it, you can't call yourself a conservative. To address your definition of a conservative, I would say that conservatives believe that in almost every circumstance, people are poor as a consequence of poor choices they made.
So, I'm one of the kind of people you can't stand. I suspect that you don't know me very well. I believe that power over one's circumstances comes from persistent personal action and not from authority. Do people end up with bad luck out of no action of their own? Of course, everyday. But is that the defining circumstance of human existence, or is persistence, faith and audacity?
Re:Conservative/Liberal take on it (Score:2, Interesting)
Unfortunately, a huge number of people who identify themselves as conservative don't seem to realize that condition++ doesn't necessarily meet condition >= decency, or even condition >= survival. It is easily, easily possible to be so low that incremental improvement through hard work is still not enough to get you to the point where you'll be on your own. There is a threshold below which basic 'modern' living is not possible. This is where having basic social services is really handy. If you can get someone above that threshold, they can often stay above that threshold on their own. It's kind of the modern equivalent of knowing how to make fire, or knowing how to fish...
It's also a shame that so few people in the US have any understanding - or put any thought into - the meanings of their tidy political categories (e.g. "liberal" and "conservative"). It's sickening how often those terms are applied in total disregard to their denotations.
Ok, that's all for now. Time for more coffee.
Personal Responsibility (Score:2)
You fat? Sue McDonalds.
You fail your classes? Sue your teachers.
You offended by something someone has said? Sue the author.
You want to maintain your manufacturing job that is an overpriced labor cost to your company? Sue to prevent outsourcing.....etc.
Re:Personal Responsibility (Score:2)
Terrorists crash into the building housing the brokerage firm your spouse worked for? Don't stop at suing the airlines, whine about not getting enough money from the government because he/she made a lot of money (and wasn't smart enough to have life insurance).
Oh, I'm sorry. People at brokerage houses tend to be conservative. I was hoping to help your argument. Looks like you'll have to find another stereotype to liberal-bait.
Re:Conservative/Liberal take on it (Score:5, Insightful)
So hard work does nothing, eh? It's all luck and circumstance. Pfff! Every time I've stopped worrking my situation got worse, and every time I started working again, it got better. What luck I must have had lately, since I've been doing better the whole 8 years I've been working where I am. I've even been lucky enough to have people pay me when I do side work for them on weekends!
Are you on crack ?
This is my favorite quote above:
People get hit by trains because they might have just plain been unlucky
Provably false. People get hit by trains because they're on the railroad tracks when the train goes by. This has nothing to do with luck, or circumstance. It has to do with bad decision making. Nobody's car ever "stalls out" on the tracks like in the movies. People try to "beat the train"... and fail.
Re:Conservative/Liberal take on it (Score:3, Insightful)
Re:Conservative/Liberal take on it (Score:4, Insightful)
8 years ago I was barely making minimum wage had a wife a kid and due to bad decision making had a degree that was basically worthless. Working two jobs and for all intents and purposes living in a cardboard box.
By making a hard choice, go into the Air Force, while in I worked hard and learned everything about anything anybody would teach me. I also looked around the world and came to realize that computers and networks were *very* big and only going to get bigger I started to insert myself into every situation where they needed someone to help out with any computer related. At the same time I started to devour every book on networking that I could get from any library that I had access to. As a result of that I got out 4 years later and worked at a crappy temp job for a few months while shopping out resumes. Since I had very little experince I got a crappy job doing tech support for a medimum sized ISP. While there I continued to work my ass off and learn everything I could and read everything I could get my hands on. As a result of doing that for the past few years I've managed to go from *very* poor to be rather well off. And it was all hard work. I have yet to meet a person, myself included, who does not waste time. Put even a small perctange of that wasted time to work and you can improve your situation. So yes they should work harder and study more.
Re:Conservative/Liberal take on it (Score:2)
You almost reinforced one of the major points the author had; the few bright spots outshine the many who don't succeed, but yet, their methods are held up as being the 'simple way' to escape a situation.
Which leaves us with:
1. Most people must be lazy, you were one of the few who could actually force yourself to do the work.
OR
2. You made it, with hard work and dumb luck. Most people don't get the dumb luck, regardless of how hard they work.
I assume you vote 1. I vote 2.
Re:Conservative/Liberal take on it (Score:2, Insightful)
I did not mean to present my case as being "simple" far from it and yes there was luck involved. My point in answer to the parent post, and granted I could have explained more, was that working harder, doing something diffrent, and taking advantage of luck when it happens, another example from my case a Cisco gawd who also happened to like me, can and almost always will allow you to improve your situation.
To me the parent seemed to be saying that working harder will not help people who are in very bad situations. I gave my example to show that working harder can help and in most cases will. So I would not call it dump luck because IMO chance favors the prepared.
And I'm not calling them lazy. But you will notice that the event that set everything in motion involved a risk and was in the short term harder than staying put would have been. People need to have the courage to change and then take advantage of the "luck" those changes can produce.
I think most people are taught that they can't improve themselves and therefore don't even try. By pointing out that in many cases you can make your luck I hope to change that perception and to make the point that no matter how bad off you are or how hard you are working at the moment you can improve.
Re:Conservative/Liberal take on it (Score:3, Insightful)
I agree with most of what you say.
I think my main beef was people always blaming the person for being taught that there was no use for trying to succeed. For me, thats something to empathize about, to try and help them out of. It's not a good reason in and of itself as for why we shouldn't care for their wellbeing. However, you seem to recognize this; I just suspect that many dont.
Re:Conservative/Liberal take on it (Score:2)
In general, I agree: many train accidents are the result of people wagering their future in the hopes of saving 3 or 4 minutes. But don't turn a blind eye to how design and infrastructure can influence things...
We used to have a very akward, high-traffic, T-shaped intersection where the train crossed over the secondary road very close to the intersection (e.g., imagine drawing train tracks parallel and just below the horizontal line of the "T"). The design was very poor, and there were several aggrevating factors with height, visibility, parking lots, etc.
One night, 4 schoolteachers got hemmed in by traffic while crossing the tracks. They were killed by an oncoming train. The driver could have been making very reasonable decisions while negotating her turn onto the secondary road... bad design is the primary culprit for that accident. Fortunately, this intersection has been heavily redone to properly handle the traffic it must support. The road is flat, everything is visible, the signaling mechanisms are obvious, and there are no adjacent parking lots to tangle up traffic flow.
The human mind is an operational hazard. Good designs take the inevitability for error into account and provide mechanisms to prevent or mitigate such errors. Designer and operator must both be serious about what they do.
Re:Conservative/Liberal take on it (Score:3, Insightful)
Way to predictably react to my statement in a way that only serves to re-inforce my point: Your perception is your truth. Check the replies, and you'll note lots of cars stall on tracks. Or get stuck on tracks through no fault of the driver.
You simply choose to err on the side of discrediting your fellow man
Re:Conservative/Liberal take on it (Score:5, Insightful)
Where did he say that? He actually said "Luck, circumstance, and opportunity play huge roles in where you are." Huge roles being the keywords. As someone else who replied to you said, how about being hit by a hijacked airplane while sitting in your office? Or being born as a woman or slave in Sudan [aol.com]? Keep working hard, maybe you won't get beaten as badly.
Re:Conservative/Liberal take on it (Score:2)
Also, when approaching a railroad crossing, you should open your window a bit so that you can hear the train even if the safety mechanisms are broken.
Exactly! Turn down that thumpin' bass you're so dang proud of and maybe look up and down the track before you cross!
Re:Conservative/Liberal take on it (Score:2)
Slashdot lameness filter (Score:2, Insightful)
Complete egoism (Score:5, Insightful)
What ruined the book for me was Nassir's incessant and shameless self-promotion. He shows the same hubris that he finds reprehensible in others and he seems to be trying to settle a few old scores by maligning people.
But the most ridiculous part of the book is when he starts drawing links between finance and neuroscience and trying to explain traders behavior based on our very preliminary understanding of brain physiology. Utter crap.
The basic message is sound: Wall Street is full of people getting paid millions who haven't the slightest understanding of math, science or technology. But that message is clear in about 10 pages and the rest is self-serving, self-promoting tripe.
Re:Complete egoism (Score:2)
It seems like personal charisma and salesmanship can become main drivers once you get to the money management level these days. Not exclusively, of course - there are a number of highly qualified and dedicated professionals who view their trade more as science than art. But the rewards are so great that the Gordon Gecko's [amazon.com] of the world are drawn to it like moths to a flame...
Re:Complete egoism (Score:2, Informative)
Re:Complete egoism (Score:2)
Since we've recently (and hopefully) completed a big Boom-Bust cycle, many of the ideas in this book have a large effect on investing. We can't just throw money into anywhere and expect a return. Nor can we throw our money at the best performing investment strategies and expect it to outperform other strategies. Survivor bias is a big part of every type of investment strategy.
Taleb's other book "Dynamic Hedging" is a must read for any quantitative finance specialist. Hopefully his self promotion doesn't ruin the book for me. This isn't a factor in "Dynamic Hedging" because of the more technical/analytical nature.
Re:Complete egoism (Score:5, Informative)
The book has one other serious flaw which really merits mention in a review: it is very badly written. 'Meandering' does not describe the book as well as 'completely unstructured'. Also, there is too much repetition for such a short book.
The overall impression is that Taleb has published the notes for a book, rather than the book itself. With proper writing and editing, these notes would, of course, make an extremely brief book; but certainly there is material here for an instructive essay.
Why are YOU alive? (Score:3, Funny)
Its my belief that some people are only alive because its illegal to kill them.
dumb luck and slashdot (Score:3, Funny)
Re:dumb luck and slashdot (Score:2)
Re:dumb luck and slashdot (Score:2)
true but, (Score:4, Insightful)
In poker, I never expect to get the cards I want, but I do stay in the game when the odds are high mathematically. If I get my flush or full house, then it was my knowledge of probablity that succedeed. If I have to fold, then nothing is lost, because the result was unpredictable, and I made the best choice according to my opinion.
So, while people often overread things (like the possiblity of terrorist attacks), we can still succeed in randomness if we have knowledge of all the variables. This knowledge, is of course, what leads to success in all fields that involve probability.
interesting book (Score:5, Interesting)
One thought - his big point with regard to the market is that people who tend to make a small/medium, consistent profit may be trading in a style which has a low-probability, high-damage outcome. This "black swan" event would/will wipe out those people when it occurs.
His big strategy is to be hedged against the worst which can happen and also make big money on the unlikely events in his favour.
Big question: does anyone know how he did when the markets crashed? (The book was written a couple of years back). If he has been practising what he preaches (and if it works!) then he should have made a killing...
Re:interesting book (Score:3, Informative)
Here's the article (Score:2)
It's an interesting idea but it doesn't really get you anywhere. Yes, there will be a disaster but you have to balance how much it costs you to be wrong alot to how much you will win when you are right (which is by his own admission unknowable). It's like a casino -- as long as you can afford to double your bet every time you lose eventually you will win big but you need deep pockets for this to work.
Mistaking cynicism for theory (Score:3, Insightful)
sad (Score:4, Funny)
Obviously trained in the Classics...
longest lucky streak (Score:2, Interesting)
Basically, whatever he did in business worked out, to the point that even when there was a disaster, like a ship going down at sea, etc., there was some lucky angle to it, and he still made money off it.
This streak lasted for his entire long life. It got to the point he would invest in weird stuff, etc. lots of things at random. All turned to the good. Darned if I can find a link, though, on short notice.
Re:longest lucky streak (Score:2, Funny)
Luck, Darwinism, Fate (Score:2)
My favorite quote from this is: Another of Taleb's peeves is the human tendency to over-attribute every random event
Not everything is related. Cause and effect are sometimes not part of the equation, but sheer randomness is. There are a lot of people out there that like to draw conclusions based on small fact... there are others who make chances based on the same.
Is the fact that the price of gas went up drastically based on the forthcoming conflict in Iraq? Possibly, maybe it's just a random thing. Perhaps it's a little of both... those that control the prices of gas raise them, knowing that many people will blame it on the Iraq problems, and thus allowing them to get away with it.
You can use probability and common-conclusion to make planned events seem random, and often - the other way - to make random events seem planned.
I meant to do that - phorm.
Re:Luck, Darwinism, Fate (Score:2, Insightful)
It's a survival feature. Humans are excellent at pattern recognition. If you hear a rustling noise and catch a glimpse of something behind you, you don't think - it's probably not a tiger, I'll wait until I'm sure.
There isn't a lot of survival related downside to seeing patterns where there aren't any. False positives are cheap.
Perceptions of coincidence (Score:5, Informative)
http://www.nytimes.com/2002/08/11/magazine/11COINC IDENCE.html [nytimes.com]
Re:Perceptions of coincidence (Score:2)
denial of history (Score:5, Interesting)
well, i suppose i'm asking for it, but
there are *alot* of cases in poker (and presumably, in investing) where one can do the "right thing" and lose big money. this is because short-term statistical fluctuations dominate in poker (investing?). the only edge one can gain is long-term, and long-term means several years of playing several days a week.
The ability to seperate the correct decision from the outcome, while still being honest about analyzing one's play, is one of the keys to successful poker.
It's like Millikan throwing away the oil drops that didn't have the right e/m
-- p
Re:denial of history (Score:2)
The prime example of someone whose theory is right, but who the market goes against, is the guy who ran the Tiger funds for years and years. (Can't remember his name.) He produced stellar returns by using value investing for ~20-25 years, but then lost his shirt in the late 90's when noone cared about value anymore. Closed down his funds in 99. Since the Nasdaq went south, he's been advising some funds that have been doing quite well.
Of course, the other example is LTCM, where they claimed their theory was right, but they were unprepared for a very unusual event. What was going on here is that their theory was actually incomplete. (Assumption of independence for markets that were actually dependent, etc.) Their stuff may have actually come out right if they could have met the margin calls.
Limited Scope (Score:5, Insightful)
This is true only in endeavours which are zero-sum for a community as a whole.
Not true otherwise.
For example, take the solving of some math problem. Like Reimann Hypothesis. It's not __necessary__ that someone __has__ to solve it. The one who does, is a good mathematician.
Re:Limited Scope (Score:2)
they do not have to zero-sum endeavours. the only requirement is that success is established using the participating community as the only benchmark. people routinely label brokers as "successful" when they lose the least in a downturn. in such situations, nobody is "winning", yet there are still some who better than average.
somewhat offtopic, but.. (Score:2)
Theories (Score:3, Insightful)
Well, yeah - that's why they are theories and not laws. The same sort of argument is made by fundamentalists against the "Theory" of Evolution. It can't be proven. It's only a theory (so let's go back to something infinitely less provable/disprovable such as the divine). Theories are meant to be correct in so much as there is overwhelming evidence toward their ideas.
The book appears to make some good points, however. (Just from the synopsis provided.) The infinitude of random factors that may cause market fluctuations makes prediction completely (probably?) intractable. In CS lingo, it would appear to be NP-Complete.
Re:Theories (Score:3, Informative)
Don't want to troll too badly
Easy mnemonic: an NP problem has a solution that can be verified in polynomial time. Clearly, lots of problems that people casually say are NP are actually far far harder.
The open question P=NP question is essentially whether there's a way to quickly (polynomially) find the solution to verify.
'nuff said.
-- p
Re:Theories (Score:2)
Re:Theories (Score:5, Informative)
The infinitude of random factors that may cause market fluctuations makes prediction completely (probably?) intractable. In CS lingo, it would appear to be NP-Complete.
Okay I am a CS theory puritan and a troll. But here is the deal.
Rant #1: Intractable problems are unsolvable problems. Even if you have infinitely powerful machines. Because our logic (the foundation of our math) has "holes" that prevent it from solving them. Halting problem (can you write a program that will verify if a program is correct)is an example. It simply cant be done.
Rant #2: NP hard problems are hard to solve. You just need lots of time. Throw in a trillion years and a very powerful computer and heck - you have a solution. Quantum computing *may* solve these NP hard problems in polynomial time. Still Quantum computing cannot solve intractable problems.
Rand #3: NP problems are not the most difficult to solve problems (among the solvable problems). In fact they are the easiest. There is something called polynomial hierachy of harder and harder problems and NP is at the very bottom.
Rant #4: Only problems that have a one bit solution (yes or no, true or false) can be NP complete. Others are "hard".
Further references to be directed at "computational complexity" by papadimitrou.
Sorry but had to take it out on someone
Re:Theories (Score:2)
Don't forget the class of problems that can be transformed into a one-bit solution. Traveling Salesman doesn't initially have a one bit solution, but it's still NP-complete rather than NP-hard because it can be thus transformed.
Another Good Book on a Similar Topic (Score:5, Informative)
Random walks (Score:5, Interesting)
Re:Random walks (Score:2, Informative)
That's a bit inaccurate in two ways:
1) Taleb doesn't claim the market is an efficient random walk - he thinks the market inefficiently prices events that have low probability because people are bad at estimating low-prob events. He capitalizes on that mispricing by buying low-prob events, mostly losing money but occasionally scoring big (and hopefully making money on average).
2) Even if the market *is* a random walk, you can still expect to make money in the long term because the stock market pays a better return on average than, say, the bond market. That's because investors demand a higher return in exchange for the extra risk due to stock volatility. But that only applies to investing for the long term, not day trading. Over short terms the extra return is completely swamped by the noise from volatility.
Re:Random walks (Score:3, Interesting)
Oh, and the above assumes any general "upward trend" in, for example, a stock market, is taken out of the data. Some markets (like currency, for example) don't have an upward trend.
Devon
Re:Random walks (Score:2, Informative)
How to beat the stock market (Score:2)
Re:Random walks (Score:2, Interesting)
You can see a reprinted edition at Amazon:
http://www.amazon.com/exec/obidos/ASIN/0471119784
You might also want to have a look at Extraordinary Popular Delusions and the Madness of Crowds, by Charles Mackay.
Surfing on the Noise Floor (Score:4, Interesting)
The stock market, day traders, etc are all dependant of this sort of thing. even though the possibily that their successes are often not distinguishable from random luck gives them all nightmares.
kinda difficult to pick out signals where they are buried in the noise floor to begin with.
Can't be... (Score:2)
Occam's Razor (Score:2, Funny)
Another good book (Score:3, Informative)
Very good read.
he has some good points, but also overlooks a few (Score:2, Interesting)
Aside from that being a bad analogy (but it sounds good if you don't know anything of time series analysis), it is also misleading in that time series analysis can be proven if it will work or not.
There is an equation (that is f'ing driving me nuts right now that I can't recall the name of it) that will tell you if your time series data is truly random or if it leads to predictability.
It has been shown over and over again that the stock market, were it a perfect market would in fact be random - but since there is human intervention that drives it, it is no longer truly random and there are non-linear patterns that are within the data.
The human brain is wired to pick up on linear patterns (to the point we see them where they are in fact not really there - it is advantageous to us to see a lion in the brush and run - when in fact there isn't a lion there, than the other way around and get mauled by a lion that we never saw).
That is more to what he speaks of, humans seeing what isn't there (there are a crapload of great books written about this - How We Know What Isn't So: The Fallibility of Human Reason in Everyday Life by Thomas Gilovich immediately comes to mind) - but the way he speaks, he also includes computational analysis in there and is wrong in his generalization.
Now it is going to bug me all day as to what that frickin' algorithm is called - the one that shows the strength of your time series data... I know it doesn't start with an A, or a Z... and I don't think there is a C there... other than that, I'm pretty much at a total blank.
Re:he has some good points, but also overlooks a f (Score:2, Informative)
Taleb mentions that the long-term winners like Soros tend to have a trait in common : they hold no pre-conceived ideas, and will swing with the market. In other words, they accept the facts as they are, instead of trying to fit them into theories.
Fooled by randomness (Score:2)
To avoid embarrassment, always check first for an expectedly large seed value before wasting any of your time.
hitchcock stock scam (Score:5, Interesting)
- mailed 1/2 the folks a prediction that stock X would rise, mailed the other 1/2 the opposite.
- threw away the 1/2 who ended up with the incorrect prediction
- repeated until there were a few folks left who had gotten 10 correct predictions (either 4 or 8 people, depending on n)
- asked the remaining folks for investments.
- split with the cash
Now imagine stock analysts all making predictions. Eventually there will be a star.Re:hitchcock stock scam (Score:2)
Re:hitchcock stock scam (Score:2)
Pretending the world is NOT random is most logical (Score:5, Insightful)
The solution is to PICK something important that may not be meaningless and random. After all, the world just might not be meaningless and random. This is kind of like Pascal's wager. If you go around believing everything is meaningeless and random and act accordingly like it is, than you gain nothing. You'd sit home and drink beers and watch Jerry Springer all day because theoretically, you aren't any worse or better off than Mother Theresa, Bill Gates, or any other "successful" person. If, however, you pick a goal or ideal to work toward and give it meaning, your life does have meaning and value. At the very least, if others around you deem your efforts worthy, you will gain the love and affection of others (which MOST people are biologically wired to need and crave). Therefore, it is possible to gain something (the pleasure of being valued) out of the void of randomness by giving your life some purpose, even if that purpose is ultimately meaningless in the big scheme of things.
So keep playing the game of life. You've got nothing to lose and only stand to gain.
I read it on Slashdot!
Re:Pretending the world is NOT random is most logi (Score:2)
But I wasn't necessarily commenting on the book. I haven't read it. But based on the review and the links here, the books seems only to confirm something I already felt in my bones: traders and economists sling a lot of BS around. But you know what? A little BS in a world of uncertainty and randomness can carry you a long, long way. Tha't why traders and economists are so full of BS!
Re:Pretending the world is NOT random is most logi (Score:2)
Wasn't there something in Ecclesiastes about this? (Score:2, Interesting)
I guess it just goes to show that there is nothing new under the sun.
estimation (Score:3, Interesting)
Something else I've noticed people being bad at is estimating ratios. Example: back in high school an English teacher asked the class what the population ratio of blacks to whites in Michigan was. Some guessed as high as 50%, but I was the only one with the correct (lowest) answer of around 10%.
I have always wondered what affected this the most. It could be that we were a suburb of Detroit, or that the local television stations' news programs may have featured more black people (scaring white viewers makes for good ratings), or that these kids hadn't travelled very far from the Detroit area, or that they were just really lousy at estimation of distances and population and had no concept of the size of the state.
Certainly, some of the error may have come from cultural bias. I didn't notice any overt racism in the school, but the specter of the "dangerous negro" was/is probably still present.
Is this still on-topic? Anyway, it seems that we need to do better in our educational system of teaching people how to get their heads around the concepts of chance and estimation. Letting any bias influence these will cause errors.
Quantifying the randomness (Score:3, Interesting)
I've long since been a believer in the "largely random" nature of the world and this puts me in mind of a wonderful piece of "research" I came across in New Scientist a few years ago.
It cut through a lot of bullshit to ask the question:
"can you predict the longevity of a system or state if you know almost nothing about it"?
i.e. life's pretty random and bothering to analyse the details will only get you so far
So how can we address, from a probabalistic standpoint, questions like:
"How long will the pyramids stand"?
"How long before my bank goes bust"?
"How long before we're hit by an asteroid"?
The reasoning went like this:
Take any "thing"/"system" etc. with a finite existance and ask the question:
"With a 95% certainty, how much time does it have left"?
For the sake of argument lets say you plant a tree, and you know nothing about trees.
Ten years later you come back and the tree is still alive and so you ask the above question about the tree.
Well, the tree is clearly alive but you don't know where "now" is in it's lifespan.
From a purely probabalistic point of view you've got a 50% chance that "now" is the middle half of it's life, i.e. you're not in the 25% of it's "youth" or the 25% of it's old age. So thats a 50% chance that it's had 75% of it's life already (middle age + youth).
Turning this into an equation, if:
x = current age
y = total age
p = probability
then: (it helps with a diagram)
x = py + (y-py)/2
or, rearranging to solve for y:
y = 2x/(p+1)
Going back to our tree then, if it's 10 years old
it's got:
95% chance of making it to 10.26 years
50% chance of making it to 13.33 years
The magic figure I keep in my head is that if you know NOTHING about something, you've got a 95% chance that it will last 1/39th as long again.
I find this little nugget invaluable when considering how much to spend on insurance or investments - cos I know/care NOTHING about them.
After all, why bother with the details, life is random
Proven with Experiments? (Score:3, Insightful)
I'm just as suspicious of claims that it's all luck as I am that it's all skill.
Is anything really random...? (Score:2)
I know I am going to be flamed for this, but this very topic is discussed at extreme length in Stephen Wolfram's book, "A New Kind of Science" [cerncourier.com]. He specifically goes on end about a particular one-dimensional cellular automata, "rule 110" (IIRC), which seems to produce randomness, but which is based off of a small handful of simple rules, and a base starting condition of a single black pixel. He demonstrates in excruciating detail how, no matter how complicated you make a system, that all such systems can be brought back to the set of simple one-dimensional CAs. He demostrates how Turing machines can be set up (via initial starting conditions - not just a single pixel) which use these same CA to perform Turing machine calculations - thus these same CA can, in theory, execute (albeit very slowly) and emulate any current computer or program in existance today. He names this "the principle of computational equivalence".
He explores at great length how systems that are seemingly random can actually be simulated by these self-same CAs, which are based on simple sets of rules - blowing away the time-held notion that complexity arises from an underlying complex ruleset. It stands to reason that given the ruleset, and a result output from the CA, one can work back to the initial starting conditions - the problem arises in that we may only have the initial conditions, figuring out that ruleset is (probably) impossible.
He explores our current methods of perception and analysis, and shows how while our current methods of analysis show that something is random, as humans using our senses we have an affinity for picking out what look to be like patterns - he seems to make the point (unless I have misinterpreted him - which is very likely) that it isn't our senses decieving us, it is our methods of analysis that are incomplete. He also presents ideas and thoughts on how we can overcome these limitations.
The book is much more than that, however - I have read articles dismissing the work as everything from a form of plagerism (at worse) to restating others thoughts (at best). I do not believe this is the case. While it is true many others in the past have played with CAs, what Wolfram has done is go that extra step, building on these ideas and bringing them all together under one umbrella of thought. He acknowledges this throughout the book.
Anyone interested in these topics and others tangent to them owes it to themselves to read Wolfram's book and come to their own conclusions. I honestly believe he is on to something, which could have profound effects in the future (perhaps far in the future, but much sooner if we read it and understand it now).
Other related links:
Collection of Reviews on ANKOS [usf.edu]
Stephen Wolfram's Web Site [stephenwolfram.com]
ANKOS Web Site [wolframscience.com]
Another Book to Check Out (Score:3, Insightful)
Non-random factors defeat traders too (Score:2, Insightful)
It is impossible to prove an unrestricted negative (Score:2)
In fact the exact OPPOSITE is true. You can never prove an unrestricted negative.
Some things are inherently impossible, like a married bachelor, or round square. Only contradictions are truly false.
Anyway, here [secularhumanism.org] is a good discussion on the subject.
Come, lets play Settlers of Catan (Score:2)
People who aren't good at statistics will get creamed. Its this way in real life too. Success is not dumb luck, its playing the odds. Its just an excuse when the loser blames the dice.
"another kind of truth" (Score:2)
That it is me boyo, deduction (which Poper used to come to his conclusions about science and theories) is a whole different way of reasoning.
As an example:
2 + 2 = 4, can be arrived at by deduction completely apart from any external stimulus.
My desk is made of wood, can only be arrived at by induction, or at least by some method relying on input to the senses.
My personal thoughts are that yes, Poper's ideas are just another form of "truth" whatever that means, but that deduction is far more fundamental than induction...
Money (Score:3, Funny)
Taleb makes a living from unforeseen events...
That's his job ? Doesn't that make a cashflow forecast a bit tricky ?
Bank Manager: Mr Taleb, you're mortgage payment bounced this month
Taleb: Oh. I never saw that coming...
Re:Aren't we bred lucky (Score:3, Informative)
Larry Niven played with this idea in the Ringworld books. Basically, overpopulation meant children were allowed by lottery
Hahaha
-- p
Re:Aren't we bred lucky (Score:4, Interesting)
I always found that absurd. Every one of us simply by fact of having been conceived has overcome odds of tens of millions to one simply by winning the sperm marathon. Then we overcame another obstacle of around three to one by surviving to birth; most don't, they fail to implant in the uterus. Luck. If luck was a psi power that could be selected for, then everyone would have it in spades. Or whatever the trump suit happens to be :-)
Re:Aren't we bred lucky (Score:2, Insightful)
My portfolio went down 28% over the last year.. how lucky is that? Oh well, money lost in the stock market just disappears. Damn Iraq!
Is it Iraq's fault that people in this country are greedy and gamble their money in the stock market? Maybe it is their fault that our weather sucks too.
"Investment" isn't identical to "gambling" (Score:2)
Obviously you don't have a 401(k) retirement account, all of which have lost money recently, even the most conservative of the funds available to most people.
Investment is not equal to gambling. Some kinds of investments are so risky that they amount to gambling, but that's actually rare.
Hopefully you're merely young and naive, but in any case, anyone who believes that investing (which includes savings accounts and Treasury notes) is gambling, and therefore foolish, merely dooms themselves to remaining in the lower class. You can't just work hard and become wealthy.
Also it's worth pointing out that the modern world, with its computers, light bulbs, plentiful food, medicine, etc, could not exist in the absence of investment.
As for events regarding Iraq, they are widely agreed to be an obstacle to the U.S. and global economy rising sharply in the near future, but no, they aren't the reason for the economic problems of the recent past. That was due to a sequence of negative events starting with the dot com bust, 9/11, Enron, etc.
Re:Questionable value of such insights (Score:2, Insightful)
BZZZT. Engineers constantly have to weigh the interest of public safety vs. the unknown implications of making various technologies available to modern man.
We'd probably be dying alot more often from medicines and technology if we didn't use the 'shit happens' realization to manage the risk/reward ratio when contemplating whether to commercialize a technology. Rule #1 about new technologies is, until we can see into the future, we'll never be able to predict with 100% what effects they will have on our planet/people. If we blindly ignored the 'shit happens' mentality, wed be doing far more damange to ourselves and our planets that we are doing now. We must always appreciate that we can't be certain what will happen in the future, so never assume that the worst possible outcome couldn't happen.
Then we weigh the benifits of the technology vs. the worth possible outcome, and our best guesses at either scenario occuring, and make a decision. But in that regard, realizing that shit happens actually plays an integral part of engineering and the commercialization of technology.