Can Bad Scientific Practice Be Fixed? 444
HughPickens.com writes: Richard Horton writes that a recent symposium on the reproducibility and
reliability of biomedical research discussed one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with science (PDF), one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. According to Horton, editor-in-chief of The Lancet, a United Kingdom-based medical journal, the apparent endemicity of bad research behavior is alarming. In their
quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world or retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative. Tony Weidberg says that the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication following several high-profile errors. By filtering results through independent working groups, physicists are encouraged to criticize. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal. "The good news is that science is beginning to take some of its worst failings very seriously," says Horton. "The bad news is that nobody is ready to take the first step to clean up the system."
Betteridge Chimes In (Score:2)
No.
follow the money (Score:2)
Re:follow the money (Score:4, Insightful)
On individual days and in individual studies the science can be protected, but you will never completely remove even unintentional bias.
Willful misrepresentation of the facts to satisfy an agenda will continue as long as humans are involved in the experimentation or in the compilation of the results.
Can bad journalism be fixed? (Score:5, Insightful)
The case against journalism is straightforward: much of the news articles, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, journalists has taken a turn towards darkness. The apparent endemicity of bad journalist behavior is alarming. In their quest for telling a compelling story, journalists too often sculpt facts to fit their preferred narrative of the world or retrofit hypotheses to fit their data.
Unlike journalists, however, science will always have to bow to reality. So, yeah, bad science practice will eventually run aground when reality hits, no matter how many epicycles one add to the model. But bad journalism will persists as long as it attracts eyeballs.
Eventually - but the lies do real damage meanwhile (Score:4, Insightful)
Re: (Score:2)
In the case of children's vaccination the medical community would be wise to co-op the language of climate change activists and label the opposition as "vaccine deniers". Shame them as anti-science and anti-medicine. Point out how the anti-vax movement's loudest voices are b-list celebrities with no expertise on the subject.
Re: (Score:3)
Re: (Score:2)
What we can observe today,with certainty, is there is link between smoking and increased rates of lung cancer. We are stil
Re: (Score:2)
Well I can link to a paper that shows that cigarette smoke contains mutagens, which means that it is directly causing mutations in cell DNA. That is unless you are going to claim that mutagens circulating in the blood stream don't actually cause mutations in cell DNA.
http://www.ncbi.nlm.nih.gov/pm... [nih.gov]
This has been known for OVER FOUR DECADES you stupid moron. Look at the data on that paper, it's 1974.
The basics are when smoking was first linked to cancer it was statistical inference with unknown mechanisms.
Re: (Score:2)
Here is a project tracking 10,000+ colorectal cancer patients over 19 + years. https://www.cancercare.on.ca/r... [cancercare.on.ca]
That project rolls into this project and shares data with 5 other registries. http://epi.grants.cancer.gov/ [cancer.gov]
There is a lot of very good
Re:Eventually - but the lies do real damage meanwh (Score:4, Insightful)
There is no vaccine for the common cold, not even for the "unusually vulnerable." There are over 200 different cold viruses. As a kid, you get lots of different colds, as you get older, you get fewer because you've already been exposed to a large cross-section of them. The next generation is going to have much bigger problems because they won't have been exposed to many of them when they were young - kids with colds are not allowed in day cares so nobody else gets exposed, nobody lets their kids play in the mud any more, everything has to be sanitized (like good old soap and water isn't good enough - you have to have an antibacterial soap).
Re:Eventually - but the lies do real damage meanwh (Score:5, Insightful)
Measles vaccine effectiveness is one that is specifically in doubt.
Having looked at this problem, I note that before and after the measles vaccine was introduced, we saw a three order of magnitude drop in US measles cases with similar declines in other countries, correlating with the introduction of measles vaccines in those countries. There's just too much of an effect to hand wave away with the assertion that the world no longer practices measles parties as much as it used to or with the other assertions you make.
Also, lab tests were developed and began being introduced at the same time as the vaccines that only verify 100/25,0000 of suspected cases. A suspect case of measles is not a case of measles. It is not even a diagnosis of measles. It is a case where doctor is covering their ass for a measles-like illness by ordering the test. There is no reason today to expect a "suspected case of measles" in the developed world to have a high likelihood of being a case of measles, especially with the extremely rare incidence of measles. There is no actual evidence here that doctors have a high likelihood of misdiagnosing measles.
You know, this stuff has been explained to you before and yet you continue with your erroneous assertions. When are you going to listen to reason?
Re: (Score:2)
Good point.
Now take a science subject, combine the journalism problems with the science problems and splash in a bit of political agenda...
Add to the mix a little bit of Group Think, and you have a very very big problem.
Maybe science went off the rails... (Score:4, Insightful)
...when we replaced the scientific method with scientific consensus?
That 99 out of 100 scientists agree one thing is true doesn't make it true - it may be, it may not be, but the number of people that believe doesn't make it so.
When the scientific community is caught 'correcting' raw data and ostracizing 'non-believers' that challenge their beliefs they undermine the public trust in 'science'.
I was taught that the scientific method welcomed challenges to accepted beliefs - a return to that position would go a long way towards reforming belief in science.
Re:Maybe science went off the rails... (Score:5, Funny)
Re:Maybe science went off the rails... (Score:4, Funny)
Your post hurts Michael Mann's feelings, and should be modded up for that reason alone. ;)
FTFY ;)
Re: (Score:3)
Re:Maybe science went off the rails... (Score:5, Informative)
Climate science is probably the most scrutinized field of science right now. And despite people saying the whole field is a crock, nothing of substance is found wrong.
Re: (Score:2)
Re: (Score:3)
You are mixing "things" and "things of substance".
Re: (Score:2)
Climate science is probably the most scrutinized field of science right now. And despite people saying the whole field is a crock, nothing of substance is found wrong.
Obviously the whole thing isn't a crock, there is just a lot of noise in the field now largely owing to it being such a hot topic and gold mine for grants and publicity. The basics like the instrumental record warming for a century, CO2 measurements increasing for a century and the fact CO2 contributes to the greenhouse effect are all thoroughly solid. That doesn't mean a horde of soft science hasn't been piled on speculating about the social impacts of potential speculative future changes brought on by thi
Re: (Score:2)
His whole hockey stick temperature reconstruction has been thoroughly rebuked by The Annals of Applied Statistics [projecteuclid.org]
Of course, others disagree with that sentiment. http://www.realclimate.org/ind... [realclimate.org]
And of course, after the original Mann hockeystick paper, a few dozen more studies have been done that have agreed with his graph.
Re: (Score:3)
Climate science is probably the most scrutinized field of science right now. And despite people saying the whole field is a crock, nothing of substance is found wrong.
Obviously the whole thing isn't a crock, there is just a lot of noise in the field now largely owing to it being such a hot topic and gold mine for grants and publicity.
And the only reason it being such a hot topic and gold mine for grants and publicity is because of the deniers who want more proof. Which they then are "skeptic" about.
Re:Maybe science went off the rails... (Score:4, Funny)
"Your post hurts Michael Mann's feelings..."
There's no need to mod it down. Mann will sue your ass off, an innovation he has personally added to the scientific method.
Re:Maybe science went off the rails... (Score:5, Informative)
Mann will sue your ass off, an innovation he has personally added to the scientific method.
Mann didn't sue anyone for hurting his feelings, or claiming he was wrong - he sued them for claiming, very explicitly, that he had committed fraud, and for calling him "the Jerry Sandusky of climate science".
Re:Maybe science went off the rails... (Score:5, Interesting)
That's part of it... But I believe the biggest problem is science fairs. Once heralded as a great way to get kids involved in science and the scientific method has been ruined by a culture of excessive safety, pandering to kids, and incompetent science teachers. First, every kids science toy has been neutered by safety culture. I'm not saying we should have kits with mercury and radioactive materials like we did in the 50s, but "science" kits where you make kitchen goo instead of actual chemical reactions is lame and boring. Kids are not fooled.
Second, the increasing pressure to pass all kids or give them participation ribbons is very present at the science fair. Many kids are forced to participate, and in many fairs judges have to assign a minimum score of "good" or some such term. I have judged at the STATE LEVEL (as in, they had to do very well at the school and county levels) and have had to assign this minimum score which was still a gift. Kids come up with buzzword laden projects and make elaborate art projects that get ooohs and ahhs from non-technical people while doing no research and offering conclusions that are demonstrably wrong. Don't believe me? Go to a science fair some time and count the number of "experiments" showing ethanol has more energy content than gasoline. There are usually a dozen at the state science fair I judge. I also wonder how many projects are done primarily by the parents who don't want their kids to do poorly.
Finally, the incompetency of science teachers... This is not applicable to all teachers, but especially in poorer areas and in under performing schools, science teachers have no science background and don't understand the scientific method. They don't understand research, citations, hypotheses, or conclusions. They don't even take the time to verify experimental results with a quick Google search. The comforting thing I've noticed from judging student science projects is that most of the kids KNOW their teachers are incompetent and bullshitted their way to a good score at the science fair. At the state level, they are completely unprepared for actual questions on subject matter by professionals in the various fields. I'm a civil engineer, and I've had to shake my head in disbelief that projects are off by an order of magnitude from what they should be and it is a shock for the student to hear that as no one has reviewed or questioned their work before the state level.
What we need is a new science fair system where teachers can mentor students on projects, but teachers don't judge projects. Projects should only be judged by people familiar with the subject matter and the scientific method. If they can't scrape together the judges, maybe the science fair needs to go away or there needs to be an active campaign to recruit and support professionals to judge school science fairs. It should be no surprise that the science fair kids have grown up to do research that panders to public opinion, are lazy, have poor citations, and are filled with self-confirming results.
Re: (Score:2)
Spoil-sport.
Re: (Score:2, Insightful)
Someone who doesn't grasp that making kitchen goo involves chemical reactions, or deliberately ignores it in order to fuel their rant... shouldn't be judging state level science fairs, or taking teachers to task for not understanding science.
A candle against the dark (Score:2, Interesting)
Banding together against the barbarians at the gate who wouldn't know the scientific method if it bit them on the arse is not "scientific consensus" - it is a defence of expertise versus wilful ignorance and deliberate lies.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
...when we replaced the scientific method with scientific consensus?
Er, no. That's like positing science going off the rails because it replaced instrumentation with data.
Re: (Score:2)
That presumes some golden era of Pure Science when no scientist ever had an ego, or an agenda, or a patron that had to be appeased, or any other motive to play fast and loose with the truth ever existed.
It didn't.
Re:Maybe science went off the rails... (Score:4, Informative)
I work in a science field. I have *never* seen "truth" or "fact" set by polling. Scientific controversy exists and disagreements exist. Researchers attempt to use carefully designed experiments with measurements to resolve those disagreements. Fringe researchers get a voice, and once in a while, an "out there" idea does pay off. I have attended many conferences where unconventional theories were presented.
However, when conveying scientific results to the public or policy makers, discussing what is consensus and what is not consensus does make sense. For that purpose, we as scientists and public trust holders shouldn't let the fringe distract us from our best understanding of the world. Because 99.99% of those fringe ideas are pretty much junk.
Science does permit challenges to existing models. But in most cases those challenges will have an extraordinarily difficult time of it. Difficult, not because of some kind of popularity contest. The current best models are there for a reason. They have survived knock-down fights with other models. They have been run through multiple experimental gauntlets. Theoretical models usually have multiple consequences and these consequences can be tested in multiple, cross-linking ways. A new model can be proposed, a new way of thinking put forth, but the burden is upon the proposer to show how this new way is consistent with all previous experiments. Fringe theories should not get a free pass because they are new or fringe.
Re: (Score:2)
Science does not deal in truth, only in models. A scientist can use the word "truth" casually and understand that it refers to a prediction made by a model (a human construct to help us approximate/cope with the universe). A layperson sees the word "truth" and imagines some inviolate deity handing down an edict.
Re: (Score:2)
Re: (Score:3)
If 99/100 scientists agree one thing is true, it's more likely to be true than the alternative backed by 1/100 scientists.
Which is beside the point. Consensus isn't about truth, it's about burden of proof.
Suppose Alice and Bob both try to make a perpetual motion machine. Alice claims she has failed, but Bob claims he has succeeded. The scientific community treats Alice's claims of failure without skepticism but it automatically assumes that Bob has made a mistake somewhere.
Does that seem unfair to Bob? Well, imagine you're a rich guy and Alice and Bob are both applying to you for a job. Bob says you should give the job to h
Re: (Score:2)
An easy example of this is when climate scientists refuse to make their raw data available to those that wish to challenge their findings.
Pretty much all raw data is available now. Much more so than in other fields of science. Besides, if you really wanted to challenge their findings, wouldn't you want to go out and collect your own raw data ?
Other notable issues arise when things like the famous hockeystick graph which clearly showed temperatures rising in advance of rising CO2
Not true. CO2 started to climb around 1800. Temperatures started to go up around 1900. https://futilitymonster.files.... [wordpress.com]
Or when dire predictions are made (No polar ice by 2015!)
That was never consensus. Maybe a handful of scientists had that date as the earliest of a range.
Re: (Score:3)
>The funny thing about that is an anthropogenic influence on global temperatures has only been possible since 1950
What the hell are you on about ? You think the age of industry didn't produce a fuckton of CO2 ? We're talking about an age primarily driven by steam engines -which burnt a lot of very dirty coal, as in a LOT.
Can it be fixed? No. Can we circumvent it? Yes. (Score:5, Insightful)
Re: (Score:2)
Tighten up peer review especially STATISTICS (Score:5, Informative)
Discovering an apparent effect should result in more research - not a rush to believe...
Re: (Score:3)
Much of the problem comes from studies being published whose data is not robust because the sample size is too small to be meaningfully significant. This needs to be headlined in the abstract if it is published at all; the best magazines should refuse anything without a decent sample size, whilst the ones further down the food chain should have statisticans on hand to ask hard questions.
This is too simplistic. In some fields you can only ever get small sample sizes because collecting data is too difficult or expensive. One example is human electrophysiology studies of brain activity: you have to get quite lucky to find the right patients. Further, the term "meaningfully significant" relates to some very thorny issues. Statistical significance is conventionally defined using a p-value and this says nothing about the size of the effect. In fact, if I do a study with a HUGE sample size then I
Grant money and politics are the problems (Score:5, Insightful)
Too many "scientists" are more concerned with the next big grant than with doing quality research. And getting grants is often a lot more about politicking and ass-kissing than making a case for why you actually deserve it.
Re: (Score:3)
Re:Grant money and politics are the problems (Score:5, Informative)
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
Re:Grant money and politics are the problems (Score:5, Informative)
Half? In many fields (like medical research) it's essentially all, and there's no "at some point." Many places offer one or two year starting faculty appointments, at the end of which you're expected to have a major grant (success rate is somewhere around 10% on those). So you better get busy writing applications. Once you're established, you better keep writing them, because now you've got a lab full of people depending on you for their livelihood.
It's well more than half their in engineering disciplines as well. I worked for a research university for two decades and know that the more successful professor/researcher spends almost all their time on grant writing, with the best ones getting buy-out of their salaries so adjunct instructors can be brought in to teach their classes while they and their grad students focus on fulfilling the needs of one grant while working on the next three or five proposals. These faculty will often teach one undergrad and one grad class and that's about it. The rest of the time they are doing project management and business development tasks with the occasional sabbatical where they actually get to do research themselves. These profs also travel a lot in order to keep connections to research collaborators at other universities, private sector companies that either benefit from their research or are supplying equipment or other needs for their research and with program directors of NSF funding areas that are either current or former colleagues. They are, basically, mini-CEOs once they get to the point where they are pulling in $1 million or more per year in grant funds.
Re: (Score:3)
Re: (Score:2)
Well I half believe that article (Score:2)
It has always been that way (Score:5, Insightful)
The problem has been compounded, though (Score:2)
For a study to be funded, it must be ground-breaking. For a study to break new ground, it must be non-obvious. For it to be non-obvious, it must be, to some degree, counter-intuitive. To be counter-intuitive,it must, to some degree, be illogical (at least from a standard perspective.)
Since scientists can improve their chance of getting funded if they are studying illogical things, there's likely going to be a strong bias toward studying things that aren't true . Some of these things will not b shown to be c
Re: (Score:3)
I think there's two other interrelated things that contribute to this.
"Big Science" these days, especially in healthcare, often involves long-term, expensive studies which take years to perform. People who commit to this mode of science make both a commitment to the field, but often to the hypothesis being tested.
To get the study funded requires basically betting your career on the validity or at least the likelihood of the validity of the hypothesis.
So, if I've bought into the hypothesis that dietary chol
Re: (Score:2)
how much money does particle physics make? (Score:3)
not as much as billions of $$$$ of some new drug that may or may not work and by the time the lawsuits come you are rich and retired
Blame political interference (Score:2)
Publish and Perish (Score:2)
But who cares, because because you gotta keep turning that endless publication crank. If you don't you might get kicked off the team.
Just visualize legions of white coated scientists chained to their lab benches/computer screens, pulling a lever to get their jolt of drugs injected directly into their veins. If they don't pull the lever often enough they'll go into seizure and break their own backs through muscle contraction
Can't be fixed (Score:3)
Can bad scientific practices be fixed?
I whipped together a quick study that shows that it is completely impossible. I'm sorry, it can't be fixed.
Re: (Score:2)
Wish I had mod points, but I'll settle for burning karma. +1, Frickin' Hilarious.
Early recognition of greatness (Score:4, Interesting)
I find that many senior professors/scientists never really accomplished anything and simply became experts in an established field further establishing that field. They are threatened by anyone who comes along and shakes the tree which might cause a few of their most rotten fruit to fall. But they are also threatened that if recognized that a truly great young scientist will come along and "steal" all the grant money that is rightfully theirs because of their seniority.
There are the rare senior scientists who encourage new and radical thinking along with making sure that credit is properly assigned (first name) but pretty much without exception these are scientists who accomplished something in their day.
I find a very common song sung by these terrible scientists is that all science is now to be done by groups. Yes groups are often required to conclusively put something new to bed but almost without exception great science had some key crack opened by some one person(or two) thinking way outside the box; not merely going through a checklist.
I have long thought that one of the reasons that so many great scientists are a bit autistic is that only this way can they ignore the continuous social pressure to conform to the groupthink that the lesser scientist would prefer they would. Whereas the more social but less capable scientists are the ones who can rise to the top on little or no accomplishments and cajole and structure the system so as to provide them with a huge cut of the grant money.
Re: (Score:3)
Citation please.
Not because I'm trying to be contrary or disbelieve you, but because I'm genuinely interested in cases where legitimate, well-conducted studies showed something established to be false and which were buried because of the potential ramifications.
I'm sure it's happened, but it starts to sound like a conspiracy theory, particularly in the absence of an example or two.
Re: (Score:3)
Citation please.
Not because I'm trying to be contrary or disbelieve you, but because I'm genuinely interested in cases where legitimate, well-conducted studies showed something established to be false and which were buried because of the potential ramifications.
I'm sure it's happened, but it starts to sound like a conspiracy theory, particularly in the absence of an example or two.
Not exactly like the parent, but an example of the established knowledge refusing to acknowledge the data in front of it's face was experienced by Mary Schweitzer [wikipedia.org]. In 1993 on a dig she was on a team that had to break a T-Rex bone open to transport it. Upon doing this she found some kind of reddish material and upon looking closer at it determined it was organic. The explanation that she had actually found some form of remaining soft tissue from a dinosaur was more or less dismissed out of hand because it's
Why should science be any different? (Score:2)
Failure should be celebrated (Score:5, Interesting)
I think part of the problem is that nobody wants to publish a paper where the experiment failed--but they should.
Failures are useful; they're not wasted time. You've almost certainly learned something from a failed experiment. Maybe you learned that the setup wasn't rigorous enough, or maybe you just learned that a certain avenue of research wasn't viable for one reason or another. I get that journals are looking for breakthroughs, but it would be so useful to read a paper in your field and find out that someone already tried the thing you're attempting, and now you don't have to fail in exactly the same way.
But that requires a much more collaborative system, and one where the community is interested in finding answers, not glory.
How do you identify when it is fixed? (Score:2)
Amateurs (Score:2)
In medical research, the problem is that most of it is run by amateurs. Medical doctors receive somewhere between no and very little scientific education, and conduct research in their spare time while not treating patients, yet in North America an MD is considered not only sufficient, but actually desirable for a "clinician scientist." There are some excellent scientists who also hold MDs, but it's secondary to their scientific training. Clinicians have very creative ideas about how to do science.
Nothing new here (Score:4, Insightful)
The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue.
That has ALWAYS been true. In fact just about the only way to make a name for yourself in science is to show that someone else is wrong about something. Einstein is famous because he showed how Newton was wrong. We put forward hypothesis, test them and (in what should be a surprise to no one) most of them ultimately turn out to be wrong or defective in some way. As a general rule that is both acceptable (to a point) and expected.
Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.
Again, why the notion that any of this is somehow new?
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivized to be right. Instead, scientists are incentivized to be productive and innovative.
Bullshit they aren't incentivized to be right. Being right is hugely incentivized. The problem is that it is hard to be right about something that is actually complicated and meaningful. So we have to break big problems up into little problems and most of those aren't consequential and many are going to turn out to be wrong or dead ends. Not every bit of science is going to be of world altering importance. Some people are doing some shady things to earn a paycheck and stay in the game but they tend to get found out in due time. Science is remarkably effective in weeding out bad data over time.
Double-blind grants (Score:2)
Double-blind studies are standard practice for studies. So why not do the same with funding? Donors to a university don't get to pick and choose which researcher or topic of research will get the money (but it has to be guaranteed to go to research and not into the general fund). The researchers' funding is allocated by some random method and they don't know in advance how much they will get if any nor do they know where the money came from.
Science vs software engineering (Score:2)
I work in psychiatry research, analyzing and maintaining the sexy fMRI neuroimaging data. I also write the storage and analysis database that we use. The database usage has been growing exponentially as data sharing projects have started and the NIH has mandated data sharing. In other words, my workload of maintaining this software system has also grown exponentially. What my PIs do not understand is that software is not at all like scientific papers. Once one of their analysts (or post-docs) writes a paper
Part of the problem is that no-one is incentivized (Score:2)
"Part of the problem is that no-one is incentivized to be right."
Yeah, bullshit. A significant mistake will permanently cripple a young scientists career, if not outright end it.
fix the motivating factors (Score:2)
This is a general problem in science (not just biomedical research). I'm a physicist, and we see the same sorts of issues.
It all comes down to how academic research is funded and judged: number of papers, number of students graduated, and amount of money raised. Inside granting agencies, this is how different research efforts are compared to determine which programs get (more) funding and who gets cut. The importance of the work, the correctness of the work, and the ethical behavior (or not) of the resea
Re:Neutrino study wasn't necessarily bad science (Score:5, Interesting)
Feynman's take: [columbia.edu]
We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.
Two more examples from Ignition! [amazon.com] by John Clark.
James Dewar (later Sir James, and the inventor of the Dewar flask and hence of the thermos botde), of the Royal Institute in London, in 1897 liquefied fluorine, which had been isolated by Moisson only eleven years before, and reported that the density of the liquid was 1.108. This wildly (and inexplicably) erroneous value (the actual density is 1.50) was duly embalmed in the literature, and remained there, unquestioned, for almost sixty years, to the confusion of practically everybody.
Bill Doyle, at North American, had also fired a small fluorine motor in 1947, but in spite of these successes, the work wasn't immediately followed up. The performance was good, but the density of liquid fluorine (believed to be 1.108 at the boiling point) was well below that of oxygen, and the military (JPL was working for the Army at that time) didn't want any part of it.
This situation was soon to change. Some of the people at Aerojet simply didn't believe Dewar's 54-year-old figure on the density of liquid fluorine, and Scott Kilner of that organization set out to measure it himself. (The Office of Naval Research put up the money.) The experimental difficulties were formidable, but he kept at it, and in July, 1951, established that the density of liquid fluorine at the boiling point was not 1.108, but rather a little more than 1.54. There was something of a sensation in the propellant community, and several agencies set out to confirm his results. Kilner was right, and the position of fluorine had to be re-examined. (ONR, a paragon among sponsors, and the most sophisticated —by a margin of several parsecs — funding agency in the business, let Kilner publish his results in the open literature in 1952, but a lot of texts and references still list the old figure. And many engineers, unfortunately, tend to believe anything that is in print.)
For years people had noted that a standing drum of acid slowly built up pressure, and had to be vented periodically. But they assumed that this pressure was a by-product of drum corrosion, and didn't think much about it. But then, around the beginning of 1950, they began to get suspicious. They put WFNA in glass containers and in the dark (to prevent any photochemical reaction from complicating the results) and found, to their dismay, that the pressure buildup was even faster than in an aluminum drum. Nitric acid, or WFNA at least, was inherently unstable, and would decompose spontaneously, all by itself. This was a revolting situation.
All of this goes to show that even well-respected scientists and engineers are not immune to bad science.
Re:Neutrino study wasn't necessarily bad science (Score:5, Informative)
Re: (Score:2)
That's exactly what scientists should be doing, re-testing long held dogma taking advantage of state of the art equipment. It's a human endeavor, so sometimes they'll make mistakes. The scientists who reported the results presented plenty of caveats, but nobody listened.
Quoting this, as it's an AC zero-karma thing that most will miss, and I don't have mod points today.
This time, AC gets it mostly right. Science is about reproducibility. Opera had a weird result that they didn't understand, so they did what they were supposed to: put it out there for people to crosscheck. For what it's worth, in an emacs buffer open in another window at this very moment, I'm putting the final touches on the paper about one of the results that WAS this crosscheck. (yeah, it's taken us to
Science is fine... Academic institutions are not (Score:5, Insightful)
"Publish or Perish", Degrees that require new original ideas, Strict hierarchy structure...
Academic institutions are culturally stuck in victorian times. So if you want to work up, get the choice projects and research, you need to publish. The more your publish, the higher the chances you will move up. Because there is so much published material, people don't read it much, so they found that they can get credit for half ass work.
Your name becomes your brand, so when you try to get a grant your name+institution you will work for will get you the grant money.
There isn't any reason why Say State University of New York Buffalo can't get a grant to study seismology, but chances are it will go to University of California Berkeley not because they will do a better job, but because of the name.
Finally institutions haven't learned how to deal with today's political climate with the attempt for breaking news. Every Hypothesis is sold to the public as a new Theory... Then if that Hypothesis is shown false (as it is common in science) then the media who may have a political slant will go and say see Science is Wrong again, just like our political stance has predicted!
Science for the most part is quite work, collaborating with like minded people, with checks and balances to try to filter out strong egos. But it has gone commercial so these checks and balances are weaken as strong egos will win out.
Science != Biomedical Research (Score:5, Insightful)
Except for medical research, I'd say most of science is the same way as particle physics: the odd mistakes which tend to get caught quickly. I don't hear of frequent retractions or contradictions by chemists, mathematicians, computer scientists, geologists or even non-medical biologists like you do frequently for medical studies. In fact it is incredibly ironic that an article written by a medical researcher criticizing the poor practices in his field is so inaccurately and carelessly written. This aptly illustrates at least part of their problem.
The faster than light neutrino claim was very soft (Score:3)
If I remember correctly, the scientists were saying that they MEASURED faster than light neutrinos, and were soliciting community aid in figuring out what was going on. They weren't confident at all of their results.
It's arguable that if they hadn't published their measurements, it would have taken a lot longer for them to have got the help which resolved the issue.
Did I just make a case for knowingly publishing results which are very likely wrong? Does it in fact boil down to simple honesty from the scie
Re:Science != Biomedical Research (Score:5, Interesting)
I agree with your general tone and statement. However it is important to note the inherent limitations of biomedical research. Generally one CANNOT do large scale studies needed to get a statistically robust result. All of physics and astrophysics generally use the 5 sigma [scientificamerican.com] discover requirement which means you have to measure the effect to 3e-7. You cannot do this with people as subjects. It is hard to do this with ANY biological subject. Many of the issues brought up stem from this.
I think much of the problem is exacerbated by the public-or-perish mentality but is even more affected by the total lack of reporting null results (when you DO NOT see anything). This skews your overall distribution. It is like not accounting for trials (because you aren't). In biomedical research they need to spend more time quantifying their trials and placing their results in the proper statistical context. Just staying that you are less likely to get parkinson's disease if you drink coffee because we asked a bunch of people isn't the whole story. How many questions did you ask? Was it 100? Did you treat all those as essentially trials?
Re: (Score:3)
We had some similar problems in particle physics with claims being made and then retracted which i
Re: (Score:3)
Research being hard is not an excuse. The difficulty and assumptions should be made clear and the analysis should take this into account.
It's not a matter of "difficulty" here as it is in "selling your results." And it's not just "biomedical research" either. I've read plenty of scientific articles in fields closer to "hard sciences" (e.g., engineering research, "real" biology, chemistry, whatever) where a fairly limited experiment and data is "oversold" later in the "discussion" section as having much greater implications than it likely does. This is particularly true in smaller subdisciplines, where researchers are struggling to convinc
Re: (Score:3)
I don't mean easier as in effort- just in the scientific sense of having hypothesis or theories that are provable experimentally. Obviously nobody would have been willing to fund the LHC unless there was good reason to believe both the experimental and theoretical rigueur existed to support it.
Try to think of a biomedical experiment on the same level that would justify the expense- you can't because the field is much too complex.
Re: (Score:3)
I don't mean easier as in effort- just in the scientific sense of having hypothesis or theories that are provable experimentally.
Building detectors and accelerators requires just as much scientific input as analysis: it is not ju
Re: (Score:2, Interesting)
"Publish or Perish", Degrees that require new original ideas, Strict hierarchy structure...
Academic institutions are culturally stuck in victorian times. So if you want to work up, get the choice projects and research, you need to publish. The more your publish, the higher the chances you will move up. Because there is so much published material, people don't read it much, so they found that they can get credit for half ass work.
Your name becomes your brand, so when you try to get a grant your name+institution you will work for will get you the grant money.
There isn't any reason why Say State University of New York Buffalo can't get a grant to study seismology, but chances are it will go to University of California Berkeley not because they will do a better job, but because of the name.
Finally institutions haven't learned how to deal with today's political climate with the attempt for breaking news. Every Hypothesis is sold to the public as a new Theory... Then if that Hypothesis is shown false (as it is common in science) then the media who may have a political slant will go and say see Science is Wrong again, just like our political stance has predicted!
Science for the most part is quite work, collaborating with like minded people, with checks and balances to try to filter out strong egos. But it has gone commercial so these checks and balances are weaken as strong egos will win out.
This reminded me of two things:
1- One of my favorite Roy Scheider lines from 2010: "Look, just because our governments are behaving like asses doesn't mean we have to! We're supposed to be scientists, not politicians!"
and
2- Dr. Jeff Hawkins, the inventor of the palm pilot and handspring lines of devices, who is an avid researcher in the field of artificial intelligence, pointed out in his book, On Intelligence, the following about his approach to his interests and career path:
"Frequently hypotheses in the a
Re: Science is fine... Academic institutions are n (Score:2)
Slight correction: Edison settled on carbonized bamboo filament (and he wasn't the first to use a carbon filament). The tungsten filament lamp came several years later, and not from Edison.
Re:Science is fine... Academic institutions are no (Score:4, Insightful)
One issue with academia is that all research must be the stated hypothesis is confirmed, i.e. a negative hypothesis result is not considered valuable. Even though the elimination of a degree of freedom from consideration for further study is one of the cornerstones of science. Instead, everyone must make something new and groundbreaking.
Re:Science is fine... Academic institutions are no (Score:5, Informative)
Papers get 'impact scored'. Based on the number of times they are cited by other papers, especially other high impact papers. Basically Google page rank for papers. If Google ever tried to patent 'page rank', scientific papers 'impact scores' are prior art. It's even done 'on the internet'.
Not surprisingly, this is also gamed.
Re:Science is fine... Academic institutions are no (Score:4, Interesting)
Right, I'm in the humanities and there is this running joke that you only need to publish one really bad and obviously flawed paper on a really popular topic, and your career is certain. It's true, one bad paper, a followup book that is even worse published at 'prestigious' publisher like Oxford UP*, and you will get cited everywhere and get full tenure within about 3 years after the book has been published. I swear I'm not kidding, I've seen this more than once.
So much for impact scores and citation indices ...
-----
* I mention this publisher because he's well respected and nevertheless publishes many bad or at least dubious books without a proper peer review. I should know, because they once contacted me, a lowly postdoc from an unknown university, to review the latest book project by one of the most famous researchers in my area. It's obvious that they just googled me, as I'm easier to find on the net than some of my more established colleagues.
Falling forward not backward (Score:4, Informative)
I agree it's not a problem. As can be seen at Retraction Watch, lots of bad science if found out and retracted. That's a good thing not a bad thing. One could ask how much of published science is made up and undetected but a better question would be how many results are simply crappy in the data or crappy in the analysis. It surely dwarfs the latter. But who cares. If the result is important it will be replicated. if it's not important then no one will cite it.
ultimately it's the well cited articles that also get vetted by reproduction. Those constitute the body of science moving forward. the rest goes into the gutter of history.
In skiing the saying is, if you fall and your fall isn't forward your not being aggressive enough. It's the same in science. People will make errors. If they weren't then then were not paying for aggressive enough research.
Re: (Score:2)
It is no surprise that academia has serious problems with the integrity of it's publications (which is the root of the actual problem pointed out here) because they have created an environment where it is profitable or expedient to be less than honest, at least in the short term, if there is one constant in life, it is that nothing remains a secret forever. Academia would do well to reward the actual merits of research that does not pan out into something groundbreaking, because like Edison, it adds to the body of research that can hep to define later research that does pan out into something novel.
This times a hundred. A negative result is still a result, but that's not how most politicians and taxpayers see it. Deans and University administration is about the same as a politician these days.
Re: (Score:3)
I agree it's not a problem. But who cares. If the result is important it will be replicated. ...if it's not important then no one will cite it... People will make errors. If they weren't then then were not paying for aggressive enough research.
The problem is opportunity cost. There's usually a big lag between research getting published and being formally replicated - or debunked. Meanwhile, grants are awarded and lots of FTEs get burned to do research based on work that turns out to have been shoddy. All that time and money could have spent on research that actually proved or disproved something. "Aggressive" doesn't really enter into this problem (lack of aggression is a better topic for a discussion on science funding priorities, not design of
Re:Climate "Science" (Score:4, Interesting)
First, the gold standard of scientific proof is experimentation.
Uh... there's a lot more to science than that. But even if we take your word for it, the climatologists create statistical models based on observable variables and fit those models to collected data. The better the fit, the more accurate the predictions.
Re: (Score:2)
The problem is that those companies are profit-driven, that means that only profit generating research is done at them. And even so it is mostly short-term focused.
Re: (Score:2)
It is not wrong, it just means that valid research that does not generate enough amount of profits is not being done, or if it is it is mostly a PR campaign kind of thing. Things like reducing famine or prosthetics for the disabled.
Re: (Score:2)
And your problem is?
Why should poor people be funding scientists to carry out research that costs more than the benefits?
Re: (Score:2)
Re: (Score:2)
No, they were illiterate in the truest sense of the word.
I had students who were unaware that books have page numbers. That's how frequently they cracked a book during twelve years of compulsory education:
I.e., never.
They couldn't read the textbooks. They couldn't read my PowerPoint presentations. They were incapable of following lab manuals -- a complete killer if you're in a systems or network administration class. They detested typing and would not accept my assertion that it's a key skill, one tha
Re: (Score:2)
From Ignition! by John Clark:
Another meeting, some years later, had more interesting results. In June 1966, a symposium on fluorine chemistry was held at Ann Arbor and one of the papers, by Professor Neil Bartlett of the University of British Columbia, was to be on the discovery and properties of ONF3. Bartlett, a virtuoso of fluorine chemistry, the discoverer of OIF5 and of the xenon fluorides, had, of course, never heard of Rocketdyne's and Allied's classified research. But Bill Fox, seeing an advance program, hurriedly had his report on the compound declassified, and presented it immediately after Bartlett's, describing several methods of synthesis, and just about every interesting property of the compound. Bill did his best not to make Bartlett look foolish, and Bartlett grinned and shrugged it off—"well, back to the old vacuum rack" — but the incident is something that should be noted by the ivory tower types who are convinced of the intellectual (and moral) superiority of "pure" undirected research to the applied and directed sort.
I think you might need data to support your assertion, since I'm sure we can play anecdote tag all day long. For the record, I have no opinion on the subject, since I don't happen to have data showing whether directed or undirected research is more fruitful.
Re: (Score:3)
My position is that fraud is not ubiquitous due to the checks and balances you appear to have failed to notice (otherwise you wouldn't be pushing such a radical view). Thus your cloak and dagger fantasy would be better used in a situation where it is easier to fake things, such as in finance.
There's a very wide void between the few frauds that are difficult to discover and a situation where your odd espionage ideas would
Black and white and negative (Score:3)
No it is not.
You are being very simplistic and also getting things backwards. It's not a case of either close to 100% fraud or 0% fraud. Reality bites when people try to prove physical things that are not real so fraud in most sciences cannot be sustained for long since when experiments are repeated reality asserts itself.
I'm not worried, it's just a dem
Re: (Score:2)
https://scholar.google.ca/scho... [google.ca]
I agree though, the world would be a better place if journalists writing pieces on science were expected to provide references.