Stock-Picking Computers 218
eldavojohn writes "A while ago, Slashdot ran an article on Algorithms used to augment or replace analysts. Today, the NY Times is running an article on stock-picking computers with quotes from the lovable Ray Kurzweil." From the article: "'Investment firms fall over themselves advertising their latest, most esoteric systems,' said Mr. Lo of M.I.T., who was asked by a $20 billion pension fund to design a neural network. He declined after discovering the investors had no real idea how such networks work. 'There are some pretty substantial misconceptions about what these things can and cannot do,' he said. 'As with any black box, if you don't know why it works, you won't realize when it's stopped working. Even a broken watch is right twice a day.'"
Efficient markets (Score:5, Insightful)
Re: (Score:3, Insightful)
Re: (Score:3, Interesting)
Re:Efficient markets (Score:4, Insightful)
Anyway, it's already being done, ergo it's possible!
Re:Efficient markets (Score:5, Insightful)
And this fact is EXACTLY what stabilizes the market!
As soon as there's a discernable pattern, somebody's going to exploit that pattern in order to make more money, and as soon as that happens, the original pattern gets interrupted, thus stabilizing the marketplace. Perfect? No. But damned good. Some regulation is needed to keep these market forces from being overwhelmed - but the cost of this regulation is a pittance compared to the benefits gained!
Money is an awfully effective invention for distributing wealth, which is why the Star-Trek "utopia" where nobody needs money is not going to happen anytime soon. So long as there is differentiation between different people (and thus resource distribution potential) there will be money.
Re: (Score:2)
People seem to forget that stock prices do, at least occasionally, reflect reality. So the "original pattern" might not get interrupted. Imagine the classical example of company that pretend to have found gold. At some point the truth comes out and the stock price of th
Re: (Score:2)
If you RTFA, this is exactly what they are doing/trying to do now.
Re: (Score:2)
Yes, but it is even more effective at concentrating wealth.
There are just so many cows and sheep you can accumulate and keep
Re: (Score:3, Insightful)
Yes, but it is even more effective at concentrating wealth.
While you probably didn't intend it, that is one of the benefits of money and banking. Back two thousand years ago, almost no one could plan ahead 20 years except the wealthy. Any savings you might have accumulated could easily be stolen. IMHO, that was one of the reasons land was so valuable. Someone couldn't break in and take it and it still had use even if everything on it was destroyed. Now, anyone can concentrate wealth, not just the extrem
Re: (Score:3, Insightful)
While this thread has passed all bounds of decency or on-topicness, I do happen to think that money still is far more effective at distributing wealth than concentration. First, money forms a small part of the actual wealth out there which also consists of real estate, labor and education, equities and other securities, savings, etc. Sure one can often trade real estate or stock directly for someething else, but these things aren't in themselves money (unless you're about to claim everything with value is m
Re: (Score:3, Interesting)
The communist ideal of Star Trek, as I see it, is possible only because of the replicator.
Consider what happens when it is possible, given an example of a given product, to duplicate it en mass at near-zero cost. Suddenly nearly everything's free. Manufacturing costs nothing, so it's all in the pattern that tells the replicator what to build. T
Re:Efficient markets (Score:4, Insightful)
Sure you can create programs that handle arbitrage opportunities, or detect shortterm effects (market movements lasting less than 1 hour), and these make lots of money for those lucky people who have realtime prices and no brokerage costs (I.e. investment banks, etc).
Stock prices for a company will move on news. Prices may drift around on speculation, but eventually a company will post its trading figures and you will know exactly how much that company is worth at that point in time. Unless these technical analysis programs know which comanies are moving product, who is about to sue who, which companies are in secret negotiations, what the future price of oil will be, etc, then they are going to miss price movements caused by events external to the markets.
Re: (Score:2)
$1 says it won't work. (Score:2)
Trader 1 (on the phone): Buy May belly contracts at...
STOCKBOT:- That's a big mistake, Sir.
Trader 2: Why shouldn't we buy now, STOCKBOT?
STOCKBOT:- The price is going to keep going down.
Trader 1: Randolph, this isn't Monopoly money we're playing with.
Trader 2 (on the phone): This is Randolph. Hold that belly order a moment.
Trader 2: Tell me why you think the price of pork bellies is going down.
STOCKBOT:- It's Christmas time. Everybody
Re: (Score:2)
Re: (Score:2)
While he cannot say the nature of the programmed trading algorithms, he does not that all their best years are when the market goes DOWN, because their algorithms are better picking
Re: (Score:3, Informative)
Um, that IS beating the market: all investments are a combination of risk and expected reward (e.g. treasuries are low-risk and low-return, junk bonds are higher risk and higher return.) If you can reduce risk without reducing return, you can make buckets of money (people will line up outside your door wanting to give you capital.)
That said, don't trust any academic studies on this topic. There are
Re: (Score:2)
While the idea of stock picking algorithms is neat; ...
The article is actually talking about at least two different different techniques:
Only the first one is really a stock picking algorithm.
Or you coul
Re: (Score:2)
I sort of thought up how one would basically create such a program. You couldn't simply write one but you would have to "evolve" it.
First take a computer program and have it randomly pick One million criteria from Google news. Then based off that criteria, randomly pick and choose a stock to buy or sell. Now repeat a few billion times. Then kill off the programs that fail to "virtually" ma
Re: (Score:2)
Re: (Score:3, Informative)
If the Efficient Market Hypothesis were true, stock pickers like Cramer should have been driven out of the market by now. Some investors do, on average, beat the market. See Warren Buffet. Now. The hard part is figuring out if your analyst is the next WB, or just some MBA who isn't too stupid and had good luck on top of not being too stupid... for the last 5 years until he regresses to the mean for the next 10 years. So. If you could write software that picked *analysts* then maybe you'd have somethin
Re: (Score:2)
You know...I've wondered why that would not exactly be a good thing to do? I mean, the balls used are physical objects, subject to imperfections. Perhaps these imperfections would select towards some balls being picked more often than others. I was thinking of this with reference to some past 'schemes' where people in casinos kept history of what numbers came up on roulette wheels....after
Well it makes a change (Score:3)
Quite futile (Score:2, Interesting)
Anyone who really knows anything about this subject wil not post. Too much going on in trading land...
Hence AC post...
Re: (Score:2, Interesting)
Re:Quite futile (Score:4, Funny)
So in this light, how am I suppose to interpret the rest of your post?
Re: (Score:2)
1) Don't try to beat the market through picking individual stocks.
2) Buy only index funds. (An index fund is a completely transparent mutual fund that just buys what's in an index, such as the S&P 500, rather than relying on some "hotshot" fund manager who uses a proprietary screening process or something like that.) After fund fees, stock index funds* beat something like 80% of activ
Seems only reasonable... (Score:4, Interesting)
However, it still can't predict things that a human can (yet). I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments. That was the one thing that the article didn't really talk about.
So I doubt we will see these "black boxes" replacing brokers, simply suplimenting them.
Re: (Score:3, Informative)
Here's [psu.edu] a guy who incorporated yahoo message boards in his stock market prediction software a few years ago.
Actually I was surprised how few references I could find to this sort of thing. Still I don't believe this is an indication that it's not happening; rather, I think market prediction is a black art because investors don't want anybody else to know
Re: (Score:2)
[Note: the following example is based on my understanding of the stock market, which is most likely wrong]. For example, say you had 1000 shares of Google, cu
Re: (Score:3, Insightful)
Completely, but it's good that you know what you don't know--you'd lose a lot less money trading that way. It's impossible to parse stock news and figure out if it's good or bad. Using your example, if Google announces that it will exceed its previously projected earnings, it's just as likely the stock will crash as skyrocket. This is because market participants are working with earning estimation at
Re: (Score:2)
I still have a hunch that it might be possible to do something in this field. Take product ship dates. While there may be occasions where announcing that you're going to let a deadline slip
Re: (Score:2)
Market theorists love to point out with much amusement that a lot of things that seem like they would be strongly associated with a clear move in one direction turn out, after you analyze them properly, to be close to 50/50 that you can't trade on them usefully.
It seems like first impressions would matter a lot in such cases. You know, simple metrics like "increased earnings == good", "layoffs == o
Re: (Score:2)
In this case, NLP means Natural Language Processing. This is the process of taking information written in a human language (English, for example) and using a program to extract useful information out of it. It's a step that would absolutely be required for the rest of my system to work, since all these announcements the computer would be required to make decisions on are wr
Spam! (Score:2)
It also can't take into account stock-pumping spam, which has graced so many of our inboxes lately.
Humans *really* bad at predictions. (Score:2)
However, it still can't predict things that a human can (yet).
People are really bad at predictions. Managed funds rarely even keep up with simple index linked funds.
I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments.
No this really should be pretty simple. We already have an algorithm that can classify a message as spam or not spam with 99.5% + accuracy, basically good or bad. The same algorithm could be used to classify any news article as good or bad for a particular stock by looking at the direction the stock takes after the news appears. You could even train the classifier automatically by looking a couple of days
Re: (Score:2)
If you e
Re: (Score:2)
A single announcement wouldn't be used that way on it's own, it's simply a way to condense a text announcement down to probably good or probably bad, it m
Re: (Score:2)
I guarantee you that if you put some serious time into this approach, you'd discover that the weight of any "announcement tone" indicator would end up so close to 0 that computing it in the first place would be a waste of time. Here's a better indicato
Technical analysis (Score:2)
If it was really possible to predict price behaviour on trading patterns these opportunities would be fully exploited until such a prediction would not work.
Why not simply buy a good profitable company?
Or a company that will be good and profitable?
Re: (Score:2)
Eventually the market will realize this. If the market undervalues a stock it leaves an opportunity for you to scoop it up nice and cheap.
Similarly if the price is above what the company is worth, go ahead and sell, but something more attractively priced.
One alternative is to buy companies that give their earnings to shareholders, then it doesn't matter if the market realizes the value, you'll get your profits as they happen.
Re: (Score:2)
Re: (Score:2)
That is an invalid assumption. The computer will only pick up solid patterns in the data if they are there to be found. If there is a solid (useful!) pattern in the market, a computer may pick it up. But the question of whether there are such patterns in financial markets is just as important (if not more so) as whether we can find them with a comput
Re: (Score:2)
If by "solid" you mean "tradable" then no, it won't. It will produce what trading system developers refer to as a curve-fit trading system. When exposed to new market data, that type of system really doesn't work well at all.
Think about this for a minute...how well would a trading system trained on data about the
This would be my request... (Score:2)
Most companies provide free trading stations but the tools/software to predict what might happen in the market cost thousands of dollars, yet these tools are never 100% accurate. So, does anyone know where one can grab Open Source versions of these? Thanx.
Re: (Score:3, Informative)
However, while it's fun to play around with a system like this, I must warn you that the realities of trading make it very hard to profit even if it looks good on paper. You probably know this, since you "got burned" before. Make sure you consult a professional before investing, or I can pretty much guarantee that you'll get burned again.
The predators and the prey (Score:2)
There are several very simple to manage portfolio strategies which will give you very good returns whether the markets are going up or going down and with little risk. It's not magic, it's common
Re: (Score:2)
Care to post a link? Thanks.
Re: (Score:3, Interesting)
Care to post a link? Thanks.
sure.
http://www.npr.org/templates/story/story.php?story Id=6203264 [npr.org]
My own portfolio is a bit simpler. UK index linked mutual fund, developing country mutual fund, government bonds, commodities (gold silver), housing stocks. Basically about 20% in each sector. Try to spread your portfolio over several sectors which don't all go in the same direction at the same time.
The strategy is simple but it's the important bit because it stops you buying at the top of the market. It's called rebalancing.
Every month add t
Re: (Score:2)
Re: (Score:2)
Of course, if you could tie your money on a small scale to the movements of the top players you could do damn good.
Sadly, this technology isn't consumer available.
Re: (Score:2)
It Works (Score:2, Informative)
Re: (Score:2)
If you have level 2 access, you can pretty accurately predict where a stock is going. Sure, you don't know from day to day, but there are reasons for market introduced delays and market makers.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:It Works (Score:4, Insightful)
Automated trading systems 'generally' are used to take a position in a stock that has already been picked.
So, trader A in Goldman Suchs wants to take a long position (buy) 100K shares of IBM, so he assigns that trade to the algorithmic trading engine, which might offer him various algorithms to help fulfill his position at the best possible price, ranging from %vol, VWAP, 'iceberg' or other type of algorithm.
notice, though, that the trader already had the stock to trade chosen, he didn't let the algorithmic engine choose it for him.
It's a prime directive issue (Score:4, Interesting)
First, no matter how well you can predict based on patterns, when you are picking individual stocks, there is such a huge influence from the chaos of human nature that, from day to day, no matter how appropriate your predictions are (based on history), they may have nothing at all to do with reality.
Additionally, if you get enough of these stock picking systems in operation, they can actually change the dynamics of the market, keeping them from being accurate for years as they all try to account for the activities associated with each others' predictions.
The problem with stocks is that in order to know how they are going to perform, you have to know not only what the company is going to do and how their customers are going to respond, but also how the investing public is going to take that news. It's an odd mix of fundementals and faith, in my experience.
The problem is markets are choice based (Score:3, Informative)
You wanna use your computer? (Score:2, Informative)
Or you could try genetic algorithms, download the info on the whole market plus historical info, give the algorithms access to the lot, plus downloaded financial news, classify the financial news as good or bad for a stock using a bayesian classifier, add that to the pot and then use evolution to see which algorithms survive best in the market to date.
You may need to build a supercomputer to run enough algorithms to perfo
Mr. Lo is not smart enough to teach at MIT (Score:5, Insightful)
[ Yes, I am joking. I'm quite sure Mr. Lo is brilliant -- just maybe a touch too honest. :-) ]
Re:Mr. Lo is not smart enough to teach at MIT (Score:4, Insightful)
You're assuming that he cares about earning lots of money. More likely, he has enough money, and he wants to do work which he finds interesting -- in other words, he's not turning down the offer because he's honest; he's turning down the offer because he doesn't want to waste 20 years of his life.
Re: (Score:2)
Wrong! I'm assuming nothing. I'm telling a fracking joke, and everyone here took it waaaay too seriously! I said, "I am joking." You even quoted me above saying, "I am joking."
OK, so I want everyone who posted a reply to my comment thinking I was serious (and that includes the moderators who modded this up as insightful) to stand up and take thi
It's harder than it looks (Score:2)
Re:It's harder than it looks (Score:4, Informative)
Re: (Score:2)
Note that LTCM had privileged market access with effective direct backing from major banks. That
The problem is too big (Score:2)
Re: (Score:2)
I hate to say this, but you give the human race too much credit. Those tens of millions of human brains tend to think in herds. Often to the chagrin of the hand full of people who sell short and make a profit.
Playing the stock market is like out running hungry grizzly bears. You don't need to faster than the bear (the market) but you do need to be faster any one else with yo
About Markets and mathematics.. (Score:3, Interesting)
The last work of Mandelbort (the 'fractals' father) 'The (Mis)behaviour of markets' http://www.amazon.com/Misbehavior-Markets-Benoit-M andelbrot/dp/0465043550/ [amazon.com] is quite interesting.
Sigh, markets are chaotic, much more chaotic than current market analisis states.
Odd (Score:2)
Re: (Score:2)
Linux and Stock-Pick Software (Score:2)
Another would be user (Score:2)
Re: (Score:2)
Buy low; sell high; make money! (Score:2, Interesting)
In theory, making money in stock market should be easy. Buy low, sell high is a guaranteed successful investment strategy and yet few people follow it. Why is that? By definition, an undervalued stock is an unpopular stock. Doing the unpopular thing is very difficult. Quite simply, human beings are a social animals and going against the crowd is often not in our best interest. Investing in unpopular stocks almost always feels wrong.
However, if you take a rational look at popular and unpopular stocks, you
Re: (Score:2)
Something that was easier to do 10 years ago then today.
Re: (Score:2)
If the price stays flat you lose against "inflation".
You need to do much more analysis than say "This stock's price is low, it's time to buy."
Vitesse (VTSS) got to over $100. You could have later bought it for the bargain price of $10. The most recent close was $1.20.
Program trading is the new day trading (Score:5, Interesting)
In the industry it's called "program trading" and refers to automated, algorithmic trading of instruments such as stocks, futures, forex. This is regularly done by many banks and large funds, and also small investors. In fact there is a discount brokerage which I'll just call IB here, that has an API which lets anyone program their own computerized trading. It's a bit "too easy" to do.
That doesn't mean it's always profitable in the long term, but without a doubt people are profiting at least in the short term. The software has multiple strategies, well documented approaches and algorithms. Generally the trading robot is trying to ride trends.
As someone who follows these things, here are a few criticisms I'm aware of:
1. These short term trading activities require high leverage, because trades have to be for large amounts of money to make them worthwhile. You need large amounts of money to make this work, because things like trading costs eat into profits tremendously. Again, like day trading.
2. High leverage is risky because one big mistake or unpleasant event could wipe out tons of past small gains. Risk management becomes a key issue. Some would argue that perceived risk in markets these days is unreasonably low. Does this unbalance the risk/reward equation?
3. Market-wide, we know program trading has increased dramatically on US exchanges. Add to this the undocumented program trading (smaller traders who don't have to report it to anyone) and basically there are a ton of computer algorithms out there today trading stocks. Everybody can't make money at the same time, so to profit the participants have to use even greater leverage = more risk.
4. Programming flaws, bugs, or improper risk management could have tremendous market-wide implications. Take for example the huge market moves in 1987; the drop was a "20-sigma" event and not anywhere within the realm of possibility back then. Obviously the models failed to handle it. Similarly, the next time we have a "big event" in markets, today's algorithms might fail. If a large number of computers choke while trading, could bad things happen?
5. So under unstable market conditions, the program trading could lead to increased volatility (like daytrading caused volatile markets during the crash). But under stable market conditions, like we have today, program trading seems to smooth out daily movements. Notice that the US markets hardly move as much as 1% in a day; trends are smooth and volatility is extremely low. The VIX, a volatility measure, has hit historic lows.
Working Examples (Score:2)
The number of events/updates comming through, at least 7 years ago, was about 800 every second. That's a lot of information.
So, what can we do with that much information that is updated so frequently?
Well, implementing time and sales for just transactions takes lots of disk space. Gi
Humans can't do it... (Score:3, Insightful)
Of course, the entire planet's GDP is only 60 trillion, so even a little mistake means complete global meltdown.
.
Re:Why? (Score:5, Funny)
No, no, no, there's a better way. Get me a case of beer and a copy of the WSJ and a dartboard. I'm at least as random as a monkey after 24 beers.
Re:Why? (Score:4, Funny)
Remember: sticks and stones may break your bones but feces just splatters.
Re: (Score:2)
On the plus side, we still wouldn't need to get you vaccinated, imported, or liscensed.
~X~
Re:Why? (Score:5, Interesting)
I had a friend who worked in the AI department of Lockheed (about 20 years ago) and they developed the software that was used for the Robert Prector's Elliot Wave newsletter. Every two years they would give the program to a couple of people to try for 6 months. These people would invest $10,000, use the program and follow the guidelines, then evaluate the results. I was privy to the outcomes of three of these tests in the mid-nineties, and the lowest was earned $15,000 and the highest was earned $36,000. These are pretty good results. (However, the stock market was steadily climbing during those years and I wasn't able to compare results with EWT competition. Still, if I was able to consistently get 30%/year on my investments...)
Back in the 70's, Dean Witter had a program called PACE. I know two people who had a system for using it that earned them over $100,000/year, and they never deviated from the program while I knew them.
Then I have a friend who is a very conservative money manager (manages a couple hundred million of other people's money), and over the period that stocks crashed (remember Enron and Worldcomm?) he only had two clients lose any money, and the biggest loss was less than %15. He claims that these programs are mostly bunk. (This guy is a perfectionist, and I bet a computer is no more disciplined than he is.)
These programs are not investment management programs. The principles of investment management are pretty simple. The best book I know on the subject is still Benjamin Graham and David Dodd's book, "Security Analysis". However, the problem is finding opportunities that comply with the principles. Systematic data analysis by computer could have a profound effect, and that's what most of these programs do.
BTW, the article mentions that profits are slowing down: In Robert Prector's book, "The Elliott Wave Theory" and in his newsletter, he sort of predicts that as information becomes more available for analysis trading will be done more rapidly on spreads that may show profits as low as 1.5%
Elliot Waves = Cabalistic Numerology (Score:2)
It could happen (Score:4, Interesting)
I support the investor's right to be as stupid as they wish. A buy and hold regulation won't do as much good as something like the Fair Tax Plan, increased personal debt reduction, a balanced budget, and reduced spending. Presumably, investments are property that an investor should be able to divest anytime. Leave the government out of it.
Re: (Score:3, Interesting)
The dollar is due for a complete collapse. China has trillions of dollars and they are seriously considering insisting on other currencies for payment, same with oil producing companies all of whom hate our guts and would love to see the dollar collapse.
If you have money to invest do it on commodities. Food, minerals, oils, water etc. The smart corporations have already
Re: (Score:3, Informative)
In fact, in the Wall St. Journal's long-running contest, the experts have out-performed the darts 29 to 21 times ( ahref=http://www.webtrading.com/issue18.htmrel=ur l2html-21923 [slashdot.org]http://www.webtrading.com/issue18.htm> ). This does not mean the experts are all that great - the score is 26 to 25 againsT the Dow Jones Industrial Average.
However, in the less-long-running contest of Wall St. Journal readers ve
Re: (Score:2)
Forget about the monkey (Score:2)
Getting the required import permits, the actual price of purchase and the care and feeding of a Monkey probably exceeds the price of an analyst's consulatation services. After all, a monkey is far more valuable and useful than, say, that crazy Cramer dude on CNBC (seems like Ballmer's long lost brother doesn't he?).
It doesn't take a very sophisticated algorithm to outperform the average analyst. I suggest a far
Re: (Score:2)
Re: (Score:2)
3.14
Re: (Score:2)
I've heard a talk on this. Don't remember where. The dude used spam-like classifier (keywords, bayes, etc.) for news articles, with `stockup/stockdown' [similar to spam/nospam] indicator (within a week of a news article).
Trained it on past articles (date published) and past stock data (yahoo). Then, just like spam filters, started classifying articles on whether they'll raise or lower a stock pri
Re: (Score:2)
There are several characteristics that, even if you take only one of them, will do better than the general market over time. One is high return on equity, which is good for several percent a year.
There's no necessity to be right 90% of the t
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Informative)