Stock-Picking Computers

eldavojohn writes "A while ago, Slashdot ran an article on Algorithms used to augment or replace analysts. Today, the NY Times is running an article on stock-picking computers with quotes from the lovable Ray Kurzweil." From the article: "'Investment firms fall over themselves advertising their latest, most esoteric systems,' said Mr. Lo of M.I.T., who was asked by a $20 billion pension fund to design a neural network. He declined after discovering the investors had no real idea how such networks work. 'There are some pretty substantial misconceptions about what these things can and cannot do,' he said. 'As with any black box, if you don't know why it works, you won't realize when it's stopped working. Even a broken watch is right twice a day.'"

Efficient markets

While the idea of stock picking algorithms is neat; market history suggest it won't work a a way to predict performance. What would be interesting is to better search for arbitrage opportunities to exploit faster than others. Of course, eventually others do the same and it becomes an arms race.

Re:

The only thing an algorithm is guaranteed to do really well is join a panic and dump shares, thereby increasing the panic.

Re:

I doubt there are (m)any algorithms that trade like this. This behavior you recognize is actually a function of the way investing is set up. The fundamental problem is that most investments occur while leveraged (so you're trading on borrowed money). If yo

Re:Efficient markets

It's quite possible that there's so much program trading going on, that you may be able to predict the market (not prfectly, but at least profitably) by effectively predicting what the program traders en-masse are doing. Not that this is necessarily any easier than predicting the mass psychology behind the markets in general, but it does as least point to the fact that much of the market moves according to very deterministic forces.

Anyway, it's already being done, ergo it's possible!

Re:Efficient markets

It's quite possible that there's so much program trading going on, that you may be able to predict the market (not prfectly, but at least profitably) by effectively predicting what the program traders en-masse are doing. Not that this is necessarily any easier than predicting the mass psychology behind the markets in general, but it does as least point to the fact that much of the market moves according to very deterministic forces.

And this fact is EXACTLY what stabilizes the market!

As soon as there's a discernable pattern, somebody's going to exploit that pattern in order to make more money, and as soon as that happens, the original pattern gets interrupted, thus stabilizing the marketplace. Perfect? No. But damned good. Some regulation is needed to keep these market forces from being overwhelmed - but the cost of this regulation is a pittance compared to the benefits gained!

Money is an awfully effective invention for distributing wealth, which is why the Star-Trek "utopia" where nobody needs money is not going to happen anytime soon. So long as there is differentiation between different people (and thus resource distribution potential) there will be money.

Re:

Money is an awfully effective invention for distributing wealth, which is why the Star-Trek "utopia" where nobody needs money is not going to happen anytime soon.

The communist ideal of Star Trek, as I see it, is possible only because of the replicator.

Re:

Yes, but it is even more effective at concentrating wealth.

While you probably didn't intend it, that is one of the benefits of money and banking. Back two thousand years ago, almost no one could plan ahead 20 years except the wealthy. Any savings you m

Re:

While this thread has passed all bounds of decency or on-topicness, I do happen to think that money still is far more effective at distributing wealth than concentration. First, money forms a small part of the actual wealth out there which also consists of

Re:Efficient markets

The technical name for this is Technical Analysis, and its a load of bunk.

Sure you can create programs that handle arbitrage opportunities, or detect shortterm effects (market movements lasting less than 1 hour), and these make lots of money for those lucky people who have realtime prices and no brokerage costs (I.e. investment banks, etc).

Stock prices for a company will move on news. Prices may drift around on speculation, but eventually a company will post its trading figures and you will know exactly how much that company is worth at that point in time. Unless these technical analysis programs know which comanies are moving product, who is about to sue who, which companies are in secret negotiations, what the future price of oil will be, etc, then they are going to miss price movements caused by events external to the markets.

$1 says it won't work.

Trader 1: It's hit rock bottom. Come on, let's buy.
Trader 1 (on the phone): Buy May belly contracts at...

STOCKBOT:- That's a big mistake, Sir.

Trader 2: Why shouldn't we buy now, STOCKBOT?

STOCKBOT:- The price is going to keep going down.

Trade

Re:

A study has already been performed to examine basic computer-stock picking based on all available (standard) market data and patterns. I forget who did the original study and am too lazy to look it up but it was covered in 'investment management' by Prof.

Re:

Don't beat the market!! **Chortle, chortle** A have a good friend who works for a programmed trading firm. They make more than 15% year-on-year, some years substantially more, regardless of the direction of the market, consistently--not on average, consist

Re:

Basically, the programs do not beat the market, but lower the standard deviation (risk) involved in trading

Um, that IS beating the market: all investments are a combination of risk and expected reward (e.g. treasuries are low-risk and low-return, junk bon

Re:

If the Efficient Market Hypothesis were true, stock pickers like Cramer should have been driven out of the market by now. Some investors do, on average, beat the market. See Warren Buffet. Now. The hard part is figuring out if your analyst is the next

Well it makes a change

to see stockbrokers being made redundant by machinery.

Quite futile

Discssing this on /. is futile...

Anyone who really knows anything about this subject wil not post. Too much going on in trading land...

Hence AC post...

Re:

>>Anyone who really knows anything about this subject will not post. ...The only sensible post I've read. I do this for a living, and have been for a while. Discussion is futile because systems traders won't say anything useful. We will say nothing,

Re:Quite futile

>We will say nothing, or we will try to mislead you.

So in this light, how am I suppose to interpret the rest of your post?

Seems only reasonable...

If you have a giant set of data, and you set a computer on it for long enough, it should be able to come up with some rather solid paterns/corelations/etc. You come up with these things, and if you don't have to wory about paying the stock broker what ever cut for each transaction (because you are the broker), then constantly trading on these thigns seems like agreat idea.

However, it still can't predict things that a human can (yet). I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments. That was the one thing that the article didn't really talk about.

So I doubt we will see these "black boxes" replacing brokers, simply suplimenting them.

Re:

I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments.

Here's [psu.edu] a guy who incorporated yahoo message boards in his stock market prediction software a few years

Re:

If I had to guess, I would say that NLP techniques are probably mature enough to take stories off a newswire and determine whether the story is good news or bad news for company X (with perhaps 80% accuracy). So, if your goal was to take advantage of jump

Re:

[Note: the following example is based on my understanding of the stock market, which is most likely wrong]

Completely, but it's good that you know what you don't know--you'd lose a lot less money trading that way. It's impossible to parse stock news and fi

Spam!

and other such real world variables into how it makes judgments.

It also can't take into account stock-pumping spam, which has graced so many of our inboxes lately.

Humans *really* bad at predictions.

However, it still can't predict things that a human can (yet).

People are really bad at predictions. Managed funds rarely even keep up with simple index linked funds.

I doubt that a computer can incorporate thigns like global news, company announcements, and other such real world variables into how it makes judgments.

No this really should be pretty simple. We already have an algorithm that can classify a message as spam or not spam with 99.5% + accuracy, basically goo

Technical analysis

But if someone else uses the same algorithm and beats you to market on your trades by even a fraction of a second your information is outdated.

If it was really possible to predict price behaviour on trading patterns these opportunities would be fully explo

This would be my request...

I wonder whether there are Open Source or Free-to-use programs to trade currencies. I tried once and got "burned" hard...almost lost everything!

Most companies provide free trading stations but the tools/software to predict what might happen in the market

Re:

The system you are looking for is Genius Trader [geniustrader.org].

However, while it's fun to play around with a system like this, I must warn you that the realities of trading make it very hard to profit even if it looks good on paper. You probably know this, since you "g

Re:

Shhhh! This is slashdot! Jeez, don't you know that anything with the words "open source" in it is automatically infallible here?

Re:

Care to post a link? Thanks.

sure.
http://www.npr.org/templates/story/story.php?story Id=6203264 [npr.org]

My own portfolio is a bit simpler. UK index linked mutual fund, developing country mutual fund, government bonds, commodities (gold silver), housing stocks. Basically about 20% in each secto

It Works

Forget quotes about neural networks, program trading WORKS. Firms like Goldman Sachs have pulled in hundreds of millions in profits with program trading and will likely continue to do so. Its about replicating human trader behaviours, except with lightni

Re:

People who say its too volatile and not logical are wrong...

If you have level 2 access, you can pretty accurately predict where a stock is going. Sure, you don't know from day to day, but there are reasons for market introduced delays and market makers.

Re:It Works

That's not the same as automated stock picking.
Automated trading systems 'generally' are used to take a position in a stock that has already been picked.
So, trader A in Goldman Suchs wants to take a long position (buy) 100K shares of IBM, so he assigns that trade to the algorithmic trading engine, which might offer him various algorithms to help fulfill his position at the best possible price, ranging from %vol, VWAP, 'iceberg' or other type of algorithm.

notice, though, that the trader already had the stock to trade chosen, he didn't let the algorithmic engine choose it for him.

It's a prime directive issue

There are so many theoretical problems with automated stock picking systems that I could spend all day working on this post. Instead, I'll ignore all but two.

First, no matter how well you can predict based on patterns, when you are picking individual stocks, there is such a huge influence from the chaos of human nature that, from day to day, no matter how appropriate your predictions are (based on history), they may have nothing at all to do with reality.

Additionally, if you get enough of these stock picking systems in operation, they can actually change the dynamics of the market, keeping them from being accurate for years as they all try to account for the activities associated with each others' predictions.

The problem with stocks is that in order to know how they are going to perform, you have to know not only what the company is going to do and how their customers are going to respond, but also how the investing public is going to take that news. It's an odd mix of fundementals and faith, in my experience.

The problem is markets are choice based

While there are probably measurable patterns. Many aspects of a market are choice based. For example, today central bankers are in a real bind. If they raise rates, it could crash housing and the whole economy with it. If they lower rates, there could be a panic out of the dollar to currencies that offer a higher return on interest rates. Once that move starts, than even investments that don't do anything at all (like gold) will rocket, making the panic even worse. In the end a human is making those choices at the helm, and a only a human can have the intuitive understanding that you better buy gold (but not on margin), even if it looks like a crappy investment. Another thing, if someone knows that the computer reads pattern X Y Z as a sell, then they might try to force the stock price to make that pattern against the market to force a computer sell and get in on it at a good price.

You wanna use your computer?

There's moneybee: http://uk.moneybee.net/ [moneybee.net] or maybe wealthlab http://www.wealth-lab.com/ [wealth-lab.com]

Or you could try genetic algorithms, download the info on the whole market plus historical info, give the algorithms access to the lot, plus downloaded financial news, c

Mr. Lo is not smart enough to teach at MIT

I hate to be so mean, but anyone who turns down a job from a fund with $20 billion dollars is just not smart. He could have milked that for 20 years! He could be pouring grant money over his breakfast cereal right now, but "oh, no, sorry, but that's not technically feasible." Fool. ANYTHING can be made to sound technically feasible for the right amount of money.

[ Yes, I am joking. I'm quite sure Mr. Lo is brilliant -- just maybe a touch too honest. :-) ]

Re:Mr. Lo is not smart enough to teach at MIT

[ Yes, I am joking. I'm quite sure Mr. Lo is brilliant -- just maybe a touch too honest. :-) ]

You're assuming that he cares about earning lots of money. More likely, he has enough money, and he wants to do work which he finds interesting -- in other words, he's not turning down the offer because he's honest; he's turning down the offer because he doesn't want to waste 20 years of his life.

It's harder than it looks

Anybody remember Long-Term Capital Management? [wikipedia.org] They were a hedge fund, heavy with experts, even had a Nobel Economics Laureate on their team. They lost 4.6 billion dollars in four months time, having to be bailed out by the big banks to prevent them from c

Re:It's harder than it looks

Found it: it was Nova: "Trillion Dollar Bet". Here's [pbs.org] the transcript.

The problem is too big

Let's see... we want to design a neural net that predicts the summed trading behavior of, oh, several tens of millions of human brains? Beyond the obvious hardware scaling problems, we must think of the software used by these brains, each of which has dif

About Markets and mathematics..

The last work of Mandelbort (the 'fractals' father) 'The (Mis)behaviour of markets' http://www.amazon.com/Misbehavior-Markets-Benoit-M andelbrot/dp/0465043550/ [amazon.com] is quite interesting.
Sigh, markets are chaotic, much more chaotic than current market analisis states.

Program trading is the new day trading

First of all, the article summary seems a bit misleading because people might think we're talking about "stock picking" as in analyst opinions/monkeys picking stocks. That's not so, this is about software driven trading. Often this is of a very short term nature, so in a way the growing use of program trading is the "new day trading" fad.

In the industry it's called "program trading" and refers to automated, algorithmic trading of instruments such as stocks, futures, forex. This is regularly done by many banks and large funds, and also small investors. In fact there is a discount brokerage which I'll just call IB here, that has an API which lets anyone program their own computerized trading. It's a bit "too easy" to do.

That doesn't mean it's always profitable in the long term, but without a doubt people are profiting at least in the short term. The software has multiple strategies, well documented approaches and algorithms. Generally the trading robot is trying to ride trends.

As someone who follows these things, here are a few criticisms I'm aware of:

1. These short term trading activities require high leverage, because trades have to be for large amounts of money to make them worthwhile. You need large amounts of money to make this work, because things like trading costs eat into profits tremendously. Again, like day trading.

2. High leverage is risky because one big mistake or unpleasant event could wipe out tons of past small gains. Risk management becomes a key issue. Some would argue that perceived risk in markets these days is unreasonably low. Does this unbalance the risk/reward equation?

3. Market-wide, we know program trading has increased dramatically on US exchanges. Add to this the undocumented program trading (smaller traders who don't have to report it to anyone) and basically there are a ton of computer algorithms out there today trading stocks. Everybody can't make money at the same time, so to profit the participants have to use even greater leverage = more risk.

4. Programming flaws, bugs, or improper risk management could have tremendous market-wide implications. Take for example the huge market moves in 1987; the drop was a "20-sigma" event and not anywhere within the realm of possibility back then. Obviously the models failed to handle it. Similarly, the next time we have a "big event" in markets, today's algorithms might fail. If a large number of computers choke while trading, could bad things happen?

5. So under unstable market conditions, the program trading could lead to increased volatility (like daytrading caused volatile markets during the crash). But under stable market conditions, like we have today, program trading seems to smooth out daily movements. Notice that the US markets hardly move as much as 1% in a day; trends are smooth and volatility is extremely low. The VIX, a volatility measure, has hit historic lows.

Humans can't do it...

You didn't think the 370 Trillion in derivatives was traded by humans did you?

Of course, the entire planet's GDP is only 60 trillion, so even a little mistake means complete global meltdown.

.

Re:Why?

First you have to get a license for the exotic pet, then you have to import it, then you have get it vaccinated and take out liability insurance in case it throws feces at a passing dignitary.

No, no, no, there's a better way. Get me a case of beer and a copy of the WSJ and a dartboard. I'm at least as random as a monkey after 24 beers.

Re:Why?

Personally, I'd be more worried about it throwing the darts at a passing dignitary, or at myself.

Remember: sticks and stones may break your bones but feces just splatters.

Re:Why?

In some cases, the result seems to work more consistently.

I had a friend who worked in the AI department of Lockheed (about 20 years ago) and they developed the software that was used for the Robert Prector's Elliot Wave newsletter. Every two years they would give the program to a couple of people to try for 6 months. These people would invest $10,000, use the program and follow the guidelines, then evaluate the results. I was privy to the outcomes of three of these tests in the mid-nineties, and the lowest was earned $15,000 and the highest was earned $36,000. These are pretty good results. (However, the stock market was steadily climbing during those years and I wasn't able to compare results with EWT competition. Still, if I was able to consistently get 30%/year on my investments...)

Back in the 70's, Dean Witter had a program called PACE. I know two people who had a system for using it that earned them over $100,000/year, and they never deviated from the program while I knew them.

Then I have a friend who is a very conservative money manager (manages a couple hundred million of other people's money), and over the period that stocks crashed (remember Enron and Worldcomm?) he only had two clients lose any money, and the biggest loss was less than %15. He claims that these programs are mostly bunk. (This guy is a perfectionist, and I bet a computer is no more disciplined than he is.)

These programs are not investment management programs. The principles of investment management are pretty simple. The best book I know on the subject is still Benjamin Graham and David Dodd's book, "Security Analysis". However, the problem is finding opportunities that comply with the principles. Systematic data analysis by computer could have a profound effect, and that's what most of these programs do.

BTW, the article mentions that profits are slowing down: In Robert Prector's book, "The Elliott Wave Theory" and in his newsletter, he sort of predicts that as information becomes more available for analysis trading will be done more rapidly on spreads that may show profits as low as 1.5%

It could happen

A HUGE amount of the market is in retirement funds or investment funds designed to be liquidated through retirement. My generation will be retiring soon, and withdrawing their funds from the market. If the USA keeps producing, it may be a good place for foreign investment that will buy the stocks being sold. However, there's about 10 Trillion dollars of off-balance-sheet liabilities in Social Security, Medicaid and Medicare out there. (If the USA was Enron or Worldcomm, the last 8 Presidents and all the legislators would be in prison today.) I suspect that the government will fund these liabilities by inflating the currency and raising taxes, making the USA attractive investment opportunity for only China. The gambling schemes, including the derivatives you talk about, will fold. The world will get hungry...invest in food.

I support the investor's right to be as stupid as they wish. A buy and hold regulation won't do as much good as something like the Fair Tax Plan, increased personal debt reduction, a balanced budget, and reduced spending. Presumably, investments are property that an investor should be able to divest anytime. Leave the government out of it.

Re:

US has been unable to support the dollar so far. All other currencies have been beating up on the dollar like it was a red headed stepchild.

The dollar is due for a complete collapse. China has trillions of dollars and they are seriously considering insisti

Re:

This is one of the most beloved urban legends of people who need an excuse for their poor luck.

In fact, in the Wall St. Journal's long-running contest, the experts have out-performed the darts 29 to 21 times ( ahref=http://www.webtrading.com/issue18.htm [slashdot.org]

Re:

Basically, investing isn't zero sum because someone is taking that money with the intent of creating more wealth than interest plus inflation. For example, issuing corporate bonds to build a new factory in Dayton, Ohio. If all goes as planned, the company