How Science Goes Wrong 316
dryriver sends this article from the Economist:
"A simple idea underpins science: 'trust, but verify'. Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17th century, modern science has changed the world beyond recognition, and overwhelmingly for the better. But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying — to the detriment of the whole of science, and of humanity. Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 'landmark' studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties. Even when flawed research does not put people's lives at risk — and much of it is too far from the market to do so — it squanders money and the efforts of some of the world's best minds. The opportunity costs of stymied progress are hard to quantify, but they are likely to be vast. And they could be rising."
Re:replication (Score:2, Informative)
Exactly. As a long term Computer Scientist, everyone in my field (Computational Linguistics) knows that published results are only good if they sound plausible and can be reproduced. This boils down to citations. Bad papers (on average) tend to have low citation counts.
Feynman said something similar (Score:5, Informative)
When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.
I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.
She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.
Nowadays, there's a certain danger of the same thing happening, even in the famous field of physics.
Re:An even worse mix: science and politics (Score:2, Informative)
The scientists are the ones who brought politics into it.
Re:Anti-science? See, now you have proof! (Score:0, Informative)
Try the same thing with an AWGer and you will get disasterous results. Point out the Phil Jones ignored FOIA requests and when it looked like he would be forced to hand over his data for peer review he had it all deleted. Point out how the predictions of the last 12 years have been completely wrong. Point out how despite being wrong and there being no warming for 15 years, the IPCC concluded their research was 96% accurate and ignored how far off their previous predictions were.
AWG has become a laughing stock to people who understand science, yet no matter how much you point it out the "true believers" still call you names because they fail at discussing the facts.
Even previous IPCC members have claimed the latest IPCC report is a joke [dailycaller.com] yet I'll be called names for point that out.
Re:An even worse mix: science and politics (Score:4, Informative)
All my mod points for the ability to edit. Of course what I meant was:
The scientists are not the ones who brought politics into it.
Re:Anti-science? See, now you have proof! (Score:4, Informative)
I'm not sure what 'publish or perish' has to do with it.
I do research. I can get funding from NIH from a well designed, well reasoned approach to learn something new. What I can't get is funding to replicate some other researcher's finding.
I'd be happy to do replication work in addition to novel research, but it's a simple fact that no one will pay for salary of lab techs, lab equipment, or reagents in order to replicate something, even if I'm willing to donate my own time.
Re:Anti-science? See, now you have proof! (Score:5, Informative)
The defensiveness now built into some fields (and here I'm thinking climate science), because of unrelenting, personal attacks does put important discussions like this into a defensive context.
And this is another bitter fruit produced by the anti-science industry, because these discussions are important to have. There are a lot of mistakes in science, but (seeming to me increasingly) there is also data falsification and fraud. [Retraction watch](http://retractionwatch.wordpress.com/) is a great website, but it makes sickening reading, and I suspect that it only scratches the surface.
I mean, sometimes, no fucks whatsoever are given [chembark.com]. How that got past peer review blows the mind. And any of these [rupress.org].
Remember this [nature.com] letter to Nature (FFS!) pointing out that 70% of the papers in one of their issues didn't say what the error bar represented. How that got past the reviewers is mind boggling. Imagining how it got past the authors requires mental gymnastics. (Since the letter, Nature articles are much better, but Peer Review is not what is catching the errors).
So, lets talk about errors in scientific research, and lets talk about scientific fraud. It's important because its rampant, and despite that there are nutjobs seeing it in their peculiar light [creationrevolution.com] lets not be put off. This conversation needs to be had more often, because the problem is dug in at the highest levels of academic prestige. [wordpress.com]
Props to the Economist for bringing this up. I'd like to see this discussed in Cell, Nature and Science. And I'd like to see credible career protection for whistle-blowers.
Re:Oxymoron (Score:4, Informative)
"Trust but verify." I hate that phrase. It's used all over the place, but it is meaningless.
Interestingly enough it's a Russian proverb [wikipedia.org] made famous in the United States by Ronald Reagan.
In any situation where verification is appropriate, trust is not.
In the specific context in which Reagan used it, it indicated that both countries would trust the other enough to begin dismantling warheads, but would verify each other's progress towards the agreed upon targets. It took trust to start the process, and verification would keep it going.
I guess people just don't know what "trust" means any more.
That's entirely possible. However, in this case I think it means you should generally trust that the experiment was done in good faith, however, if you need to actually use the results, you should verify them first.
Re:Anti-science? See, now you have proof! (Score:5, Informative)
Here's a clue: when was the last time you delayed publication (of eminently publishable results) to run some extra tests, or perform alternative forms of verification? I've never had a supervisor allow such things in my entire career. It's always a case of publishing as soon as possible (i.e. as soon as a study has the remotest chance of getting past reviewers).
I tend to be very cautious in my approach to things, and I've often wanted to do additional verification work. Not to target a better journal or a second publication, but just for the sake of more solid conclusions. I'm never allowed to do this, and I even recognise that it's not part of my job to cause any problems over it, for the very economic reasons that you mention. This bothers me deeply, but it doesn't seem to bother the kind of people who care more about their careers than about the veracity of their results. I've even been told on a few occasions that my reticence to publish some of my own simulation work that "should already be out there" is bad for my career.
In my perception, those who are more career-driven have an advantage in gaming the system. They are rewarded for publishing multiple papers of shallow scope and relatively minor significance; spreading what should be presented once as thin as possible across multiple publications. We all know it's a game to be played; that those evaluating our early-career performance really have no clue whether a publication is important or not. By the time they find out, those who've gamed the system well will already have tenure.