Gambling Could Reveal Which Scientific Studies Are Worth Their Salt (sciencemag.org) 63
sciencehabit writes with this interesting story about using prediction markets to weed out false positives in scientific experiments. From the article: "Gambling is frowned upon in many circles. But what if the gamblers are researchers betting on how each other's experiments will turn out, and the results are used to improve science itself? A group of psychologists has found that their collective gambling—with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses. They propose that this type of gambling setup, known as a prediction market, could become part of how science gets done."
the interesting part (Score:5, Interesting)
The interesting part seems to be that at least scientists use different weights between asking their opinion (such as a poll) and gambling on outcomes with real money.
Re: (Score:2, Interesting)
I wonder if... these psychologists can replicate experimental results of *this* experiment... or are their experiments as good as gambling?
Re: (Score:3, Interesting)
It might be that all that is needed is some kind of limited resource so that people who make consistently bad predictions have their impact weighted down until they no longer affect the combined prediction, and that the wisdom of (thus suitably selected) crowds does the rest.
Re:the interesting part (Score:4, Insightful)
Not really. Gambling is seldom limited by if the gamblers can afford it or not.
The big difference is that usually when it comes to making predictions people have a situation where they one outcome is beneficial for them even if it was just a case of "It would be nice if this was the case."
There simply is nothing to gain from having a correct prediction and nothing to lose from having a wrong prediction so people pick the outcome based on what they would like to happen.
Bringing in money, real or fake, is sufficient for people to override this mental block and go with what they actually think, not what they want to think.
Re: (Score:2)
The interesting part seems to be that at least scientists use different weights between asking their opinion (such as a poll) and gambling on outcomes with real money.
Nope. Spot the difference:
" their collective gambling —with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses. "
Peer review commitees (Score:3)
Proposals are gambling. When you write a white paper at a company it goes through layers of managerial review if it would be worth the companies time to have you complete a full proposal. Then the Government review panel looks at and decides if they want to invest in that. Peer review informs on the odds foliding in many predictors of success, such as past performance.
Finally the reviews are parimutual gambling. Your odds of winning are not strictly the value of your proposal but the relative odds amongs
Re: (Score:2)
Even the most devoutly religious would not actively wager money to put their faith to the test. The clever ones would probably quote some scripture abo
Re: (Score:2)
Well said. There's definitely a subtle alteration of perspective going on when you put a monetary loss into the equation. I think you've nailed it by suggesting that wagering diminishes the "wishful thinking" influence on a belief.
Re: (Score:2)
> Even the most devoutly religious would not actively wager money to put their faith to the test.
People do this kind of stuff all the time.
Many people wager their lives on their belief in a God, and... usually die.
George R. Price famously gave all his possessions to the poor; got evicted, fell into depression and then killed himself.
The Future of Science (Score:1)
WTF are all these MBA's doing in my lab?
Re: (Score:2)
Re: (Score:1)
No, but I do think MBA's will start to show up anywhere they smell money.
They'd probably start offering derivative products to the scientists to mange their exposure.
Re: (Score:2)
Re: (Score:2)
Where do you think the money for those sub-prime mortgages came from?
Central banks and a relaxing of the regulations on reserve amounts (to 50 to 1 leverage) actually.
It was the derivatives.
They could have held US treasuries and still screw up at 50 to 1 leverage.
Re: (Score:2)
"there is nothing wrong with derivatives."
There are plenty of problems with the derivatives that were being cooked up in the early '2000's.
A "Credit Default Swap(CDS)" was basically a fraud perpetrated by companies like AIG. It was a way of selling "insurance" without calling it "insurance" and thus evading the capital requirements for traditional insurance policies. You bought bonds from XYZ Inc and want "insurance"(a CDS) in case they don't pay you back. AIG is happy to oblige, even though AIG doesn't
Re: (Score:2)
Re: (Score:2)
Perhaps you'd like to place a wager on that?
Fiat scientists (Score:1)
Kinda like InTrade (Score:1)
Las Vegas casino overs will have something to say about this. But better than mere gambling, let's put these guys in the ring. Make it into a Gladiator contest. Then you'll have a market.
Well if it works for business... (Score:3)
or science could weed out bad science (Score:1)
if it's actually science, then you can repeat the experiment. we solved this problem, it's called the scientific method.
It does no such thing... (Score:4, Insightful)
It's just enforcing consensus - It doesn't mean the science will be right or wrong anymore than which horse wins the race.
Yeah, the odds increase the probability of a success but that's not a guarantee.
Geez, is this what's become of "science" in the world today?!
Re: (Score:2)
Of course, empirical verification has to be the deciding factor. But right now there is very little way to communicate the insights that many scientists have about which studies are reliable. We could be much more efficient in doing future studies if there were a way to collect assessments of those who work in closely related fields without taking years to get to know people well enough to get them to tell you their actual opinions even though they haven't been able to focus on that question to publish a
Re: (Score:2)
To me, this seems like an important development. Keep in mind that there's huge biases in scientific publishing in favor of interesting studies (translation, in favor of studies with "surprising", aka unlikely, results; also against repeating and confirming experiments). Furthermore, once a journalist gets ahold of it, every piece of research is likely to produce a cure for cancer, fusion reactors, or revolutionize some aspect of computing. This based on the part where the scientist said, "hint, hint, need
Re: It does no such thing... (Score:2)
It's called a "prediction market". It's been experimentally found to have better success in getting close to the truth than other techniques we have.
Gamblers (Score:2)
Model it after fantasy football, you pick your team of scientists and win or lose based on how they do.
Draft kings is on the job as we speak.
already have this (Score:2)
We already have institutionalized team based gambling to help decide which science to pursue. It's called "grant review committees."
Re: (Score:2)
Grant review committees usually don't have much at stake when making their decisions.
Re: (Score:2)
Really? Grant review committees I've been on handed out a couple million dollars worth of bets... I mean, grants.
Re: (Score:2)
Yes, they hand out millions of dollars worth of bets... with other people's money.
And what happens to the grant committee members if the projects don't deliver anything useful? Basically nothing.
Of course, scientific research projects rarely "fail" anyway because grant recipients have an interest in representing their work in the best possible light, and neither granting agencies nor grant committees have an incentive for challenging them.
Re: (Score:2)
Scientific research projects rarely fail because even negative results can be useful scientific information.
Gamblers 2 (Score:2)
I see they have been having difficulty getting participants.
Good.
Re: (Score:2)
Yes, nothing like difficult to prove scientific theories! Being difficult to prove is working well for global warming/climate change, so why not for this as well!
All hail how making things harder helps science!
I wrote "I see they have been having difficulty getting participants."
Umm, "difficulty getting participants" != "difficult to prove." Thanks for the Jeremiad though, cowardly denier.
Yeah, but we're talking about psychology (Score:2)
A group of psychologists has found that their collective gambling—with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses.
That might work for something like psych, where few results are reproducible - or even where few "experiments" are re-tried. However, I doubt that the real world reflects the wishes of some profs with a penchant for a flutter and too much free time.
Now if they could apply quantum mechanics to the problem, instead of gambling, there might be some interesting results.
Re: (Score:2)
"Gambling is a losers game."
In Vegas, yes, because the odds always favor the house. In the real world, the odds might be in your favor. You read a paper describing an experiment and the results. It's up to you to determine if it makes sense and if you think the results can be replicated. Asking yourself "Would you bet money on it?" just puts it in a different perspective. In our culture, bringing money into the equation puts a new twist on things. I think that asking that question is simply analyzing
God does not play dice (Score:1)
I've used this for IT project analysis (Score:5, Interesting)
Estimating confidence in the completion date for an IT project can be hard. When I've been asked to review a troubled IT project, I've found that a very useful technique to determine how far along the project actually is -- and when it's actually likely to be done -- is to ask those involved at different levels what their "$1,000 confidence level" is -- that is, what date they would be willing to bet $1,000 out of their own pocket that the project will actually be done by. I find that even the most optimistic engineer suddenly turns cautious and starts thinking of all the things that could go wrong.
I then ask what their $10,000 confidence level is. Those answers tend to be the most accurate. ..bruce..
It worked for a lost nuclear weapon (Score:2)
Always a good question (Score:2)
"Would I bet money on it and with what odds?"
enumerated vs unenumerated (Score:2)
Better Peer review? (Score:1)
I have a feeling they're doing better science because if they bet against one of their peers, they know if they have skin in the game that they're going to be a lot more critical and call bullshit easier. When it's a dolt/dumbass that's handing out public money, he doesn't care. Let's see, followed outline, has some good looking graphs, isn't disproving Man Made Global Warming - don't care, give 'em money.