How Science Goes Wrong 316
dryriver sends this article from the Economist:
"A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."
Anti-science? See, now you have proof! (Score:1, Insightful)
Aaaand this is exactly the kind of thing that young-earth creationists and climate change deniers will jump on to show that science (and scientists) can't be trusted.
This is a real problem and conflict of interest (Score:5, Insightful)
Money (Score:5, Insightful)
Assuming TFA's numbers are correct, I'd bet that much of the problem is that no agency, be it government or commercial (and particularly commercial) wants to spend it's money seeing if published results are reproducible. Additionally, no one ever won a Noble Prize for excellence in reproducing others' results. Verification of results is key to science, but this is one of several aspects of doing science right that the funding agencies either don't want to, or can't (as in Congress looking over the shoulders of managers at the NSF), pay for. Everyone wants "everything, all the time" without paying for it, and this is the sort of thing that happens when decisions are driven by the money people (who may be scientists, to be fair) and not the people who know what the hell is going on.
Re:Anti-science? See, now you have proof! (Score:4, Insightful)
No. Actually it isn't. Because homeopathy can't even replicate their own results in controlled environments.
It SHOULD, however, be a wakeup call to scientists all over that their chosen fields are more caught up in the "publish or perish" mentality than they should be.
Between this, and others willing to take these unreplicated (and possibly unreproducible) studies as "Holy Writ", what people think of as science IS becoming as sloppy as religion.
Which makes it harder for the people who actually DO the grunt work and the follow-up to receive their just due.
A simple idea underpins science? (Score:4, Insightful)
We must be talking about different sorts of science, because from what i know, the simple idea rather is
"be objective, and be sure to keep it falsifiable
Re:Anti-science? See, now you have proof! (Score:5, Insightful)
Instead, if you really want to talk to a young earth creationist (I don't know why you would), you need to show them the evidence. Really dig deep. If they want to discuss carbon dating, then dig in and show the evidence we have of why carbon dating works. Eventually, if they are willing to go along with you (and it will take a lot of work so they might not), they will turn into an old-earth creationist.
And you will absolutely learn something along the way. Never turn the discussion into an argument about "the guys I trust" vs "the guys you trust" because that argument is never won, by either side.
Re: Anti-science? See, now you have proof! (Score:5, Insightful)
If you would be so kind as to replicate the catastrophic prediction results -- oh, wait, those can't be tested?
We're in the process of testing them right now. We should have results in 50 or 100 years although chances are we don't have to run the full experiment to see where it's heading.
What makes you think the results on CO2 sensitivity are "out-of-you-ass guesses" rather than just an expression of the uncertainty of the results? Where have you seen a scientist that links every bit of bad weather to AGW? There are some non-scientists who may do that but that's not science.
Re:Sounds Like Work... (Score:5, Insightful)
It's not about being lazy. Feynman famously addressed this [youtube.com] in his "Cargo Cult Science" rant in his Caltech commencement address given in 1974. (There's no recording AFAIK, that link is to someone reading the transcript).
He makes very good points: funding is for new results. Attempting to repeat another scientists published work is not a new result (unless you can't), and many places won't even allow you to try, unless it's something very sexy like observing the Higgs boson or something. It's an important structural problem, and it was worth calling attention to forty years ago.
There's no doubt that some unscrupulous researchers have noticed this and are gaming the system. The incentives to do so are particularly high in biochem.
Re:This is a real problem and conflict of interest (Score:5, Insightful)
But you get no funding from it and potentially make an enemy who now DOES have reason to scrutinize and point out your every mistake. If you aren't accomplished enough yourself your failure to replicate something isn't likely to even be published. And it isn't to the same degree. This is Dr. So-and-So, the man who did that brilliant work and discovered x,y,z impressive sounding thing vs This is Dr. So-and-So, he's never actually accomplished anything but he did a great job of failing to replicate his peer's results.
You'd be better off in the long run pretending to replicate or even expand on the results of your peers. It isn't like they are ever going to call you out on it, you've made an ally AND made it much more difficult for either of your reputations to be harmed by a third party regardless of their claims.
"And your academic progress is hampered if someone shows your results to be flawed."
Yeah, but apparently it's not likely and you can select areas of study to minimize the probability. Even if someone fails to replicate your results it isn't proof that you faked them.
"I think you are ignoring the competitive element."
I don't think so. Most people are probably working from the assumption that work accepted and that has passed peer review is most likely legitimate. Why spend all that time and effort in hopes someone else in wrong? And in a way you can prove? Even if you suspect they faked something, that just means you are likely to be able to get away with it too and as stated above there is more glory down that path.
It's no different than essays and other academic papers. You are required to provide references to support your assertions and credit sources but everyone knows the professor doesn't actually have time to read them. So people find credible and uncontroversial sources on topics that could well be saying something that could support their assertions. On the slim chance you were caught in it, you'd just find something you accidentally misinterpreted and be a little cautious for the next couple. And that's if you had to say anything, the professor is far more likely to assume (s)he has better comprehension of the topic than the student and add a note to educate the poor fledgling, they might not even reduce the grade over it depending on the topic.
Re:Greed (Score:5, Insightful)
Re:Peer review stretched to its limit by money (Score:4, Insightful)
So... you think science used to be better? Really?
Newton spent much of his energy in later years in a brutal smear campaign to smear mathematicians and scientists who in fact invented much of he took credit for, such as portions of Calculus. Edison is known to have mounted an equally brutal attack on his arguably more inventive peer, Tesla. Have you ever read Penis envy? Really? That guy was a world class crack-pot, IMO.
I've read many technical and scientific papers every year since about 1982, and I see zero degradation in professionalism. The truth is there was never much anyway. For ever paper that made me believe something I useful, there were a half dozen total crap papers that weren't even close to the mark. Science is just fine... just the same crap as always, but overall very effective crap. It's the freaking "news" networks that have turned into crap.
Re:The other issue with much of modern science (Score:4, Insightful)
It's resource intensive, but also just plain difficult. For example, publications are never a full description of an experiment, just the highlights. It takes a skilled researcher to fill in the gaps and then a second level of skill to accurately carry it out.
Looking at it from another perspective, ignoring scientific developments which are the result of inspired genius (which I would argue are rare), every new publication is the more novel and difficult work that has been conducted to date. If it weren't, it would have been done already.
So how can you expect someone else (who wasn't able or interested to carry out the work themselves) to immediately duplicate cutting-edge work based on an incomplete description?. It's a bit amazing that up to 50% of publications could be replicated at all.
Re:Sounds Like Work... (Score:5, Insightful)
He makes very good points: funding is for new results. Attempting to repeat another scientists published work is not a new result (unless you can't), and many places won't even allow you to try...
"If I have seen further it is by standing on the shoulders of giants"
Even though funding is for new results, to get new results, these days you will almost always need to build on what has gone before. So while scientists generally don't attempt to replicate published results, if the work is important, someone will eventually think of a way to extend the work, and rely on it to build something else. At that point it will become obvious if the original research is flawed.
So good science does eventually win, it just can take a longer time than people would like to spot frauds. Science works, the last century is a testament to that.
Re:Anti-science? See, now you have proof! (Score:5, Insightful)
Yes, laudable in concept. But I'd like to point out that you can't fix stupid, and trying invites heartbreak and consists of a massive waste of time. Also, you can't fix faith -- it strikingly resembles stupid in form, effect, and depth of infestation. And it's worse in one way: Being stupid is not politically correct. Exhibiting faith is. Woe is us.
argh (Score:4, Insightful)
Wishful thinking. When Galileo's presentation was found to be at odds with religion by the ultimate arbiter of such things (the then-Pope), they certainly were in conflict. The "apology" took centuries to come around, far too late to help Galileo. It wasn't a misunderstanding. It was stubborn clinging to myth and nonsense with the added salt of the power to enforce the myth over the facts.
When religion gives us social rules, there may be, often is in fact, value to be had. When religion tries to tell us how the world came to be and why we are here, it falls flat on its face, each and every time. It's the purest form of conflict: The intentional and irresponsible promulgation of fictions in the face of repeatable, consensual facts to the contrary. The more we understand, the more visible this is.
Re:Anti-science? See, now you have proof! (Score:4, Insightful)
*Ignorance* is temporary. Stupid is forever.
Re:evolution: cold, hard fact. (Score:5, Insightful)
I'm glad you used a question rather than a statement, because if you had stated what you're trying to imply, you'd be making a directly untestable and unscientific claim.
I'd prefer to keep the discussion on science, and proposing causal exclusivity to "evolutionary" processes is not science, it's a hopeful non-sequitur and inappropriate generalization. "Evolution occurs", is science. "Only evolution occurs", is not. The fact you only care about the second form, for personal reasons, has nothing to do with science or a scientific usage of "evolution" or "genetic".