The Journal of Serendipitous and Unexpected Results 153
SilverTooth writes "Often, when watching a science documentary or reading an article, it seems that the scientists were executing a well-laid out plan that led to their discovery. Anyone familiar with the process of scientific discovery realizes that is a far cry from reality. Scientific discovery is fraught with false starts and blind alleys. As a result, labs accumulate vast amounts of valuable knowledge on what not to do, and what does not work. Trouble is, this knowledge is not shared using the usual method of scientific communication: the peer-reviewed article. It remains within the lab, or at the most shared informally among close colleagues. As it stands, the scientific culture discourages sharing negative results. Byte Size Biology reports on a forthcoming journal whose aim is to change this: the Journal of Serendipitous and Unexpected Results. Hopefully, scientists will be able to better share and learn more from each other's experience and mistakes."
This could be good (Score:2, Insightful)
This IS a great idea, and the cause of our failing (Score:1, Insightful)
If we ONLY publish and peer review our successes, our failures and errata is discarded.
In this data could be a LOT of really amazing potential, given peer review and continuance.
No wonder we don't have a theory of everything yet, we're not looking at nearly all the data.
Fantastic idea (Score:5, Insightful)
Sometimes talking to people with very pro-sciene views, you get the idea that "science" is either an accumulated set of known facts or a perfect method which, because of peer review, is infallible at learning absolute truth.
In reality, it's just a set of processes that we've developed and which has been generally more successful at producing helpful results than other methods. No reason to think that the way we go about it couldn't be improved. I can't imagine that failing to share the results failed experiments doesn't sometimes result in the loss of important information.
Coincidentally I just saw this talk [ted.com] which raises the question whether helpful data can be gathered even if it's not gathered through conventional rigorous scientific methods. It seems like an interesting idea-- they're essentially gathering lots of data from various sources and using statistical analysis developed by economists to try to draw conclusions. My biggest concern would be purposeful manipulation by someone with an agenda.
But anyway, all of this is to say that this has gotten me thinking about how the scientific process may still be open to some innovation.
Problem is (Score:5, Insightful)
The problem with any change or reform of the publishing system is that publications are so important for the individual scientist. A paper isn't just a neat way to disseminate results. They are your work evaluation and your CV; they are keeping the score as it were. Where you publish and how often you publish directly determines where - or if - you work another year or two down the road.
And even a short paper takes a lot of time and effort to write. For an informal "don't do that; we tried and it didn't work"-email to a colleague you could just jot down three or four paragraphs after lunch. Make a paper out of it and you have weeks or more of work ahead of you - looking up other previous published reports on the same kind of experiment; doing your best to figure out and explain the exact causes; square your (lack of) results with the apparent success of other groups that did something similar; make neat, clear graphs and illustrations as needed; get formal permission from your lab and your funding agency (and your co-authors labs and funding sources) to actually publish the thing. Then revise and edit the paper multiple times after comments from your co-athours and reviewers.
So, getting good publications is vital for your ability to make rent and buy food for your family. Writing publications take a lot of time and effort - time that is pretty limited. So, even though the will to spread the word on a negative result may be there, chances is, writing it up will be relegated to the "when I've got a bit of spare time"-pile, where it will likely sit until well after retirement.
Awesome! It's about time! (Score:4, Insightful)
This is a fantastic idea! It takes a great deal of strength to do this; one has to learn how to have fun and ignore the pangs of the ego.
James Burke's Connections [wikipedia.org] was based on similar philosophy. Non-linear thinking is a very powerful method of moving through time. Many geeks live in the clutches of an obsessive desire to control everything so that they don't get hurt by being wrong. If they could just relax and roll with the ups and downs and not be so hard on themselves, not care if they are laughed at, then they would find their power and perhaps start living lives of consequence.
One university professor described an enormously powerful way of doing research; When you're up against a wall, seeking fruitlessly to find a specific title to continue your line of thinking, instead just pull out some random book nearby. Doesn't even have to be from the same shelf or Dewey code. It will have the answer. -But only if you're tuned to your inner Jedi.
Those who deny their inner Jedi are forever lost. But the upside, I guess, is that nobody will laugh at them.
-FL
This is a great idea! (Score:5, Insightful)
As a scientist, I can also tell tales about how the scientific method gets distorted by ideology. When I was in grad school, I was working on a complex set of problems that were a horror -- a week doing eight hours a day pumping numbers into a scientific calculator is not my idea of fun. However, back then, it was a necessary evil. So, I was about to have to do another horror week with the calculator, which I did not want to do, so I was wasting time and did something silly. It turned out to be a great idea. It gave a whole new method to solve the problem type at hand. A number of other people had a hand in the final paper, but I got to be first author. Unfortunately, as only one author amongst many. The paper made claims about the hypotheses that was being tested, I objected very strongly to this -- there was no hypothesis, but we just got lucky. However, there is a paper with my name on in, published in the 20th Century, that contains claims about what we discovered which are false, at least with respect to hypotheses and all that stuff, in order to ensure that we were following someones idea of the scientific method. It irks me even today. Fortunately, a book about the issue now gives a more accurate account. However, there is no doubt that scientific ideology can drive out the truth. Thus, what is proposed here is a good idea. Telling the truth (even if it does not conform to the ideologically driven official method) is something I teach my grad students even today.
Great scientists weren't very scientific (Score:5, Insightful)
Einstein wasted the last half of his life on wishful thinking "God does not play dice". Well turns out we're pretty sure he does. See Bell's theorum which shows that it can't just be hidden variables. And by all accounts for a theoretical physicist he sucked at advanced math.
Isaac Newton was a horrible little man. Ill tempered, neurotic, and did wild experiments that he was lucky didn't blind him. Let's not forget the nastiness with Leibniz.
Galileo had the social skills of a village idiot which led to the suppression of his work and his imprisonment by the authorities that he angered. (They were idiots too but that's beside the point)
They're three of the greatest but I could go on.
We like to pretend our scientists are great men with a couple of eccentricities that are way too smart to socialise or tolerate fools but the fact is their thinking isn't so superior OR logical OR scientific EXCEPT in their areas of expertise. THAT is why they are remembered. Not because they were above being unscientific.
Re:A great idea (Score:3, Insightful)
Egos are massive and competition is fierce, so asking researchers to admit a mistake or give the competition a short cut is a tall order.
The funny thing is, discoveries are not "I told you!". They're "That's interesting...".
Re:Fantastic idea (Score:3, Insightful)
This is because statistical error -- the error you make because of the limits of your sample size -- goes down as 1/sqrt(N), but systematic error -- the error you make because of imperfect knowledge of your model, biases you can't estimate and control for, and so on -- does not.
A large and difficult part of the field of computational physics I do, at least, is accurately estimating systematic errors. Statistical errors are easy, you just do the probability shit. But honestly estimating systematic errors is hard.
If you have a large sample size, of course, you should be trying to bring that huge sample to bear to reduce systematic error, which can usually be done. An example of trying to correct for systematic error is the corrections made to polling data to account for cell-phone-only voters. It can be done, and as long as it is done honestly, you will get a reliable estimate of significance levels at the end.
Old Problem (Score:3, Insightful)
Trouble is, this knowledge is not shared using the usual method of scientific communication: the peer-reviewed article. It remains within the lab, or at the most shared informally among close colleagues. As it stands, the scientific culture discourages sharing negative results.
This sort of complaint goes back a very long ways, and it's certainly as good a time as any to address it head on.
We have a habit in writing articles published in scientific journals to make the work as finished as possible, to cover all the tracks, to not worry about the blind alleys or to describe how you had the wrong idea first, and so on. So there isn't any place to publish, in a dignified manner, what you actually did in order to get to do the work, although, there has been in these days, some interest in this kind of thing.
- Richard Feynman, Nobel Prize Acceptance Lecture, 1965
Re:This is a great idea! (Score:3, Insightful)
One of the things that many scientists lack, is a good grounding in the Philosophy of Science. The public version of science, largely pushed by science teachers has an origin in the Vienna Circle of Logical Positivists. This is now largely known to be problematic, but is still the prevailing view.
Thanks so much for pointing this out. It never ceases to amaze me how many scientists seem to believe blindly in some sort of simplified method taught in middle school or the interesting, but ultimately useless, Popperian epistemology.
Is it really that complicated to understand that "falsifiability" is only a useful concept (and a somewhat limited one at that) for describing the process of testing hypotheses that are already formulated, but it gives almost no guidance about how to come up with such hypotheses in the first place? With such a "method," how could scientific progress ever happen?
Such a "method" is not supported by any reasonable empirical study of the history of science. It's sort of ironic that with all the data available about how scientific advances actually seem to work, scientists believe in a paradigm of their own discipline that doesn't describe the evidence.
Re:This IS a great idea, and the cause of our fail (Score:3, Insightful)
Re:A great idea (Score:3, Insightful)
And why you should get funding to do a follow-up study.