Slashdot Log In
Most Science Studies Tainted by Sloppy Analysis
Posted by
ScuttleMonkey
on Tue Sep 18, 2007 11:22 AM
from the devil-is-in-the-details dept.
from the devil-is-in-the-details dept.
mlimber writes "The Wall Street Journal has a sobering piece describing the research of medical scholar John Ioannidis, who showed that in many peer-reviewed research papers 'most published research findings are wrong.' The article continues: 'These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. [...] To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.'"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Oh yea baby... (Score:3, Funny)
Yup. (Score:3, Insightful)
I came here for the global warming ref (Score:3, Informative)
And I wasn't disappointed. You also managed to please me with your inclusion of the bogus "The Great Global Warming Swindle", "Global Cooling", and conspiracy theories. Excellent!
As an interesting aside, I thought that this argument had been dropped because it was a little too easy to shoot down:
The interesting thing is that, despite warming temperatures, the oceans are holding more CO2 than before (which lowers their pH level as CO2 + H20 = C2H03, ca
Re: (Score:3, Insightful)
Just to be somewhat relevant - while there are negative feedba
Devastating the world's economy (Score:4, Informative)
Parent
Re: (Score:3, Insightful)
The CO2 lag (Score:3, Informative)
According to the scientists, the program misrepresented what they said.
>the concept of CO2 warming was a fairly small area of research that wasn't taken very seriously
On the contrary, it goes back to Arrhenius and is generally agreed to be the reason the oceans aren't frozen over. The existence of a "greenhouse" effect was in science textbooks decades ago.
>CO2 levels rose about 800 years AFTER the temperature rose.
After the temperature BEGAN to rise. Temperature and CO2
Re: (Score:3, Informative)
You probably don't care, so let's go to the refutation of your claim:
From Wikipedia [wikipedia.org]: "Carl Wunsch, professor of Physical Oceanography at MIT, was originally featured in the programme. Afterwards he said that he was "completely misrepresented" in the film and had been "totally misled" when he agreed to be interviewed.[23][5] He called the film "
Re:Yup. (Score:4, Insightful)
Parent
Here it comes... (Score:4, Funny)
as a phd student (Score:3, Insightful)
Fairly common knowlege (Score:5, Informative)
1. A good number to shoot for is 15 journal articles in your first 6 years. If you don't have tenure in 6 years chances are you are never going to get it. The point of being published is to get the name of the university out.
2. Should be self-explanatory. You need to bring in $$$ to the university. The more you bring, the more profitable you are and the more they need to keep you around. But publishing is still more important.
3. Teaching, while as students we all feel is important, is actually the least important thing towards tenure. A mediocre or even bad teacher who writes papers (that get accepted by excellent journals) at a rapid pace will get tenure where an excellent teacher who can't write for the life of him will not. This is why you often see people from industry teaching. They teach for the love, tenured professors are there for the research and for the higher level teaching (where it is more a relation of facts, not an educational process).
The 'sloppy analysis' referred to is not 'fraud' as you cite. There is a difference between fraud and sloppy analysis. The rush to put out papers (between 2 and three a year, by this guide, for tenure) causes some slop to occur. As a reference, I've been working on a paper with my advisor and a (yet-to-be-tenured) professor for almost a year already, and we are just submitting it to a major journal. And the paper is based mostly off of my thesis work completed a year ago! A good paper and good research takes time. But please, do not mistake sloppy analysis for fraud. Mistakes are one thing, deception entirely another.
SOURCE: Advice to rocket scientists: A Career Survival Guide for Scientists and Engineers. Dr. Jim Longuski, published by the AIAA in 2004. But again, this is fairly common knowlege and can be found anywhere you look. As a postdoc (I am too) I'm suprised you didn't know
Parent
Sensationalist... (Score:5, Insightful)
Re:Sensationalist... (Score:5, Insightful)
Parent
Sloppy analysis (Score:3, Insightful)
Re: (Score:3, Interesting)
Medical research vs. basic research (Score:4, Informative)
Re:Medical research vs. basic research (Score:5, Interesting)
Parent
Re: (Score:3, Insightful)
Out of curiosity:
1) What is the usual failure rate for replication?
2) Do the letters routinely get published?
3) You just do that for work you're following up with experiments, not for ever
What About this Study? (Score:5, Funny)
According to my research (Score:4, Funny)
My experience (Score:3, Insightful)
Well use the Scientific Method then (Score:5, Insightful)
And one of the first rules is, "Never take a single study as proof of anything! Wait till the results are replicated before you even think of moving to a conclusion."
The major problem is really poor reporting on science research. The news media routinely blazon some **NEW * Scientific * Discovery!!!**. Then you read the story and somewhere around the 10th paragraph you might see that this is based on only one study - and oftentimes even before peer review.
Every scientists knows this. It's a shame the public doesn't. They wouldn't worry so much.
Slashdot taints article study finds (Score:4, Funny)
Authorities are also investigating the connections between Malda, Bush Laden, Bill Gates, Dvorak and Borat [mit.edu] SCIgen is a program that generates random Computer Science research papers, including graphs, figures, and citations. It uses a hand-written context-free grammar to form all elements of the papers. Our aim here is to maximize amusement, rather than coherence.
o the irony (Score:4, Funny)
pot, meet kettle
"Most science..." (Score:5, Insightful)
His work seems to focus on population genetics and epidemiology, which is notorious for having unreproducible claims due to a combination of uncorrected multiple testing, publication bias and statistical incompetence. This "gender and genes" is a perfect example: someone does a study, finds nothing, slices and dices the data until he gets p = 0.04 for females or Asians or smokers and publishes his breakthrough finding. I'd have been surprised if he hadn't found almost all of those to be wrong.
If you look at more in-vitro molecular biology and biochemistry work, I doubt if nearly as high a percentage of it is clearly "wrong", although quite a bit of it is worthless.
Re: (Score:3, Interesting)
And as math goes, statistics can be pretty darn counterintuitive.
(I speak from experience - I worked for a few years in a university computer center's "academic support group", where among other things I help faculty and grad students with running statistical analysis packages. Some of the experimental designs were pretty bad, too.)
Re: (Score:3, Interesting)
Actually you need more math for a PhD in psychology or sociology than you do for molecular biology, unless you're in a really math-heavy specialty.
The biologists, being generally smart, can usually pick up what they need (math, programming, equipment building) on the fly. The problem is that, as you say, statistics frequently is counterintuitive.
Double checked (Score:3, Funny)
No Money in Replicating Results (Score:4, Insightful)
That being said, it's my understanding that most scientists work off of grants, and those grants fund novel research. Replicating results is of obvious importance in validating those results, but doing so seems at odds with the funding mechanisms that are the reality for what I would believe to be most researchers.
Are researchers supposed to replicate the experiments of others in their spare time and on their own dime?
(As rhetorical as that might have sounded, I actually welcome those with first-hand experience to respond to it)
Bad Dates (Score:3, Informative)
It must be so... (Score:3, Insightful)
TFA
Since the criterion is that the claim is published, someone had to find the study new and interesting. Most new ideas are going to be wrong, especially true the more significant it is. After all, how many crackpot theories were postulated between Newtonian and Relativistic physics? On the other hand, most things easily verifiable, etc, are too obvious to me considered new and interesting. Note, while I find this interesting, I did not come up with this idea. Some economists published a similar study over a year ago postulating this as a reason. Of course, it's probably wrong.
strong variation with fields (Score:4, Interesting)
Cargo Cult Science (Score:5, Insightful)
Feynman spotted them over 30 years ago. He called them Cargo Cult Scientists [wikipedia.org]. They put on the appearance of science, but have none of its substance. They give a good performance, like an actor playing a scientists on TV. They wear the clothes, speak the language, seemingly apply the methods. But it's all empty. There's no rigor. There's no insight. There's no real testing going on. It's all just people waving around graphs, and lines, and their qualifications, and formulae they don't understand, to support the theories they want to be true, regardless of whether they are true or not.
It's because in this day and age, you can't be a witchdoctor. You can't appeal to spirits, or gods, or karma, or any of the other philosophical reason thrown up in past ages. We live in "The Age of Reason", and people expect things to be proven to them "scientifically". So all the people who in the past would have risen high by browbeating, appealing to authority and writing great prose, are forced to dress themselves up in white coats and go through the motions of an experiment before they proclaim their great revelations to the world. The experiments however, are just as empty as all the old techniques, and bear only superficial relation to actual science.
Personally, I think it's gotten worse over the last 30 years. The unwillingness of actual scientific communities to challenge the misapplication of their methods by unscientific ones has lead to a dilution of the authority of science as a whole. Under the current regime any half baked psychiatrists can show pictures to 20 undergraduates, record a few squiggles on an MRI, run the numbers through R over and over until he gets what he wants, and proclaim to the world just about whatever he likes, and still be called a scientist! No wonder it's all too easy for the Intelligent Design movement to pose as "real science". Just look at how low the threshold for real science is.
There's only one way to deal with Cargo Cult Scientists. You have to call them out. You have to show how flimsy and false their supposed science really is. You also need to learn all the old rhetorical techniques, because faced with someone who actually knows what they're doing, the Cargo Culter will fall back to very old and time honored methods which enable him to win from a weak or false position. I think the real scientific community owes it to itself to show up these charlatans for what they really are, Con men. If they don't, science will just become more diluted in the long run until the public regards it in the same way it regards homeopathy.
PhD != Research Scientist (Score:4, Interesting)
Here's my problem. Only a fraction (I'm guessing 1 out of 5) are actually capable of doing good research. The rest are competent employees for developing other people's research into useful products, but aren't terribly original thinkers, nor show a lot of initiative, nor show the rigour and clarity of thought one wants to see in a researcher.
Frankly, when I "unleash" employees on open-ended problems without much guidance, the majority soon begin to flounder.
There is nothing wrong with getting advanced degrees, but many then feel they are obliged to do original research when in fact they really aren't up to it. This may be one reason why the quality of papers isn't where it should be.
Flawed medical studies != all of science (Score:3, Informative)
I'd expect the rate of error for physics experiments to be much lower than that of, say, sociology.
I'm sorry (Score:3, Funny)
a pointless statement (Score:3, Insightful)
There are almost no areas of science we're "done" with. The most recent paper on a subject almost always points out where previous papers have gone wrong. Thus, the previous papers have some mistake such as a miscalculation, poor design or incomplete analysis. If you pick any paper published in a peer reviewed journal this month, there's a very high probability that at some point in the future it will be amended or improved by some other paper.
What Ioannidis *has* shown in his recent reports is that in genetics, not enough people are publishing on the same subjects. There are not enough "other papers" out there to check on the previous ones. The result is that papers which in other fields would be recognized as needing improvement are instead treated as the final word.
Summary (Score:3, Funny)
and
Most WSJ Articles Tainted by Sloppy Analysis (Score:4, Insightful)
But Dr. Ioannidis has a very narrow definition of science - he only includes statistical studies that use p 0.05 as a threshold for significance. There are, of course, lots of papers that do not show p-values - the purification of a protein, the determination of a genome sequence, the identification of a new fundamental particle. In many cases, p-values are not provided because they are not considered informative - something that happens when the p-value is much much much less than 0.05 (I like my p-values less than Avagadro's constant. With that p-valuep, I think most of my results are correct.)
And, of course, the WSJ misses all of this. The point of the research paper is that you can do everything right, and still be mislead with marginal p-values (0.05). Not sloppy, just not significant enough. We could, of course, require more stringent values, but then we would miss the genuinely rare, but important results.
As the research article points out, results that are reproducible are, in fact, quite likely to be correct. It is perhaps useful to distinguish between science as a paper and science as a process. Most results that stand up to scientific scrutiny over a period of years (that any one cares enough about to validate), are (probably) correct. In some disciplines, which rely heavily on modest thresholds for statistical significance, many results cannot be confirmed.
Uncomfortably close to the truth (Score:4, Interesting)
I have worked in a biotech / pharmaceutical environment for over five years now, and I don't trust the average medical researcher or biologist to accurately calculate the weight of a kilogram of stuff. I would say that their data analysis is habitually poor, if I were not convinced that it is actually habitually awful.
I have been trying to change this for five years. My success in this has been such that it contributed strongly to my recent decision to start searching for another job. The reality is that biomedical researchers simply do not believe in doing mathematical analysis of data properly. They consider it an eccentric habit, forgivable but socially objectionable, like smoking. By common consensus, it is considered much too complicated to expect that any of them can be expected to understand. Your average biologist is innumerate to the nth degree, and proud of it.
I blame their education, which seems to stress naive and antediluvian (excuse the word) analysis practices, if at all. I have seen course materials which in their expression of basic mathematical formulas, betrayed that they had been left unchanged since the days when people used slide rules and logarithmic tables for calculations. Most of their other training is strictly qualitatively, not quantitatively, and focussed more on memorizing that on understanding.
If necessary, they will find a crutch to help themselves to stumble along: Find a paper that defines a formula that looks relevant, and then fill in the numbers. They would not bother doing their own analysis, or trying to understand how the calculation works or whether it is relevant at all. The notion that a good statistical analysis of mathematical modeling can actually contribute to the scientific understanding of an issue, is well beyond most of them.
I am frankly, sick and tired of their attitude, and I still have to work with these people every day. And in my experience, my colleagues are actually better than most. I strongly suspect the WSJ is correct on this one.
"Do Scientists Cheat?" (Score:4, Interesting)
`Abstract: This video examines the troubling question of scientific fraud: How prevalent is it? Who
commits it? And what happens when the perpetrators are caught? Factors contributing to "bad science"
include sloppy research, personal bias, lack of objectivity, "cooking and trimming", "publish or perish"
pressure, and outright fraud. The limits of peer review and other quality control systems are discussed.'
The results of the study determined that 48% of all published data was fraudulent. The data was trimmed, cooked or outright falsified. Some cases made famous by public exposure were analyzed.
While recieving a lot of lip service from the establishment science, the two government researchers who made the report were reassigned to worthless tasks in isolated areas. One was sent to shuffle papers in Alaska, IIRC. So much for whistle blowers, even government whistle blowers.
In the last 19 years it seems nothing has changed. Besides this latest report how can I tell? Simple. The news is filled with stories of drugs being recalled because they are more dangerous that the problems they are supposed to treat. How would they ever have gotten on the market in the first place if their FDA "studies" weren't rigged? And you don't wonder about the revolving door policy between Pharmaceutical employees and FDA employees? Corporate influence in research is as corrupting as Microsoft influence in ISO standards voting.
What really burns me is that MUCH of our basic research is done at academic institutions by professors funded by government grants, i.e., tax payers. But, thanks to the best congress that money can buy (because most of them have been bought off) OUR research is "monetized" (sold to special interests) for pennies on the dollar. These interests then reap HUGE license profits for decades. To make matters worse, many of the "special interests" are the very academic researchers who were paid to do their work. Having discovered key facts, without reporting them, they resign academia and begin a corporation to capitalize on what we paid them to learn.
IF we had a congress worth what they are paid there would be a law which prohibits recipients of gov grants, or their families relatives, or former business associates to personally benefit from what they learned using that grant money for a period of 15 years. Secondly, the ONLY corporations which should be allowed to receive IP licenses from the gov should be NON-PROFITS, whose board, management or employees cannot include the professor or his family or relatives.
Another thing that this recent study shows is what the NOVA film revealed: Peer-review is worthless for vetting research. Replication is worthless for vetting research. Obviously, personal integrity is also a worthless indicator of research quality.
Re:How is this news? (Score:5, Funny)
After all, studies show that most studies are wrong.
Parent
Re:How is this news? (Score:5, Insightful)
Clever.
The fact is, good science is hard work. In fact, it is damn hard work, requiring not only a supremely keen intellect but a very high tolerance for tedium, great attention to detail, and usually a big fat wad of cash. Also, it requires a profound lack of ego (and the ability to cope with failure and keep trying), given that a trememdous amount of effort could (and frequently does) wind up being completely discounted by a peer-review or another study.
The endeavor of scientific research obviously provides us tremendous benefits, and is furthering the evolution of our species at a blindingly fast rate (depending on how you look at it, of course). It is very important, very hard, and very expensive.
There are many, many people who would like to be scientists but really don't have the brain for it (as I stated above, it isn't just intelligence that matters). Unfortunately, a lot of them wind up doing research anyway, and they cause problems. Hopefully there are enough good scientists with enough funding to clean up their mess.
Parent
Excellent contribution to the discussion! (Score:3, Informative)
The media often contributes by being dishonest and over-interpreting results.
Most "scientific" papers aren't really scientific. The first clue is that they are poorly written, suggesting that the writers want to hide their poor contribution behind bad expression.
Slashdot editors often are fooled by "junk science", I notice. For example, this article was fraudulent in my opinion: Imaging Breakthrough "Sees" Lung Disease [slashdot.org].
The Slashdot artic
Re:How is this news? (Score:5, Insightful)
Parent
Re:How is this news? (Score:5, Funny)
You'd think a postdoc would have known this.
Parent
Re: (Score:3, Interesting)
If I had the patience, I'd write two versions of each one of my papers, but th
Re: (Score:3)
Re: (Score:3, Insightful)
Personally, I think it's true. Between publish-or-perish, financial conflicts of interest, and various political and social movements trying to influence science, a lot of published science is worthless.
However, there's a difference between believing that and actually showing that. Anybody wanting to fix that needs clear and convincing proof first.