The Neuroscience of Screwing Up 190
resistant writes "As the evocative title from Wired magazine implies, Kevin Dunbar of the University of Toronto has taken an in-depth and fascinating look at scientific error, the scientists who cope with it, and sometimes transcend it to find new lines of inquiry. From the article: 'Dunbar came away from his in vivo studies with an unsettling insight: Science is a deeply frustrating pursuit. Although the researchers were mostly using established techniques, more than 50 percent of their data was unexpected. (In some labs, the figure exceeded 75 percent.) "The scientists had these elaborate theories about what was supposed to happen," Dunbar says. "But the results kept contradicting their theories. It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."'"
Ridiculous (Score:4, Interesting)
"It wasn't uncommon for someone to spend a month on a project and then just discard all their data because the data didn't make sense."
That doesn't mean the data is wrong, it means the /hypothesis/ was wrong, if not the theory, and needs to be modified.
If they're really throwing out date just because it 'doesn't make sense', they're doing religion, not science.
You never discard the data (Score:5, Interesting)
Re:Ridiculous (Score:4, Interesting)
And this is what bothers me. If you are willing to run an experiment enough times, you will eventually get data to support your assertions. Get a statistical 90% certainty, and it could be that you ran the scenario 100 times, and throw out the 99 times that did not give you this certainty. The scientific process is bullet proof. The folks who "do science" not necessarily so.
Good! (Score:5, Interesting)
If problems occur as you postulate elaborate hypothesis, then stop piling up the elaborate hypothesis! But be sure and still make available your existing (complex) hypothesis, methodology and unexpected data - preventing others from going down the same path with the same methodology is still highly valuable!
Let's say you're looking at a production and consumption cycle involving neurotransmitters and neuroreceptors of some sort, and the various channels of input and output involved. Your starting presumption you base your hypothesis on is that there is a buildup which triggers an electrical signal to stop consumption and clear the channel. The only evidence you can realistically gather for now is protein density at a certain output channel - but others have worked to ensure this is a reliable approach specifically under these circumstances.
So, you do the specific experiment, trigger the signal, but you get a wildly different result - the stop in consumption occurs, but the protein density does not change at all in the output channel. What actually happened is still unknown, only you haven't verified any correlation with your hypothesis. You still have valuable data, but no mechanism to verify under the circumstances. Either your methodology failed, or you misunderstood what was happening - and the world of knowledge is made larger by either... even if your paymasters won't get happy about the result.
Science is often like throwing pebbles in complete darkness - it takes a lot of stones and close listening to make out a mental picture of the scene - especially when there's a lot of noise already around. Everyone would love it if we could just flip the lights on - but we have yet to invent a light that can see into the inner workings of the functioning brain very well. Gotta keep throwing those pebbles for now.
Ryan Fenton
Re:Why most scientists and engineers screw up (Score:4, Interesting)
The dirty little secret is that the Y is not always unexpected, just too politically incorrect and dangerous to be released to the public.
So, when reality is racist, you change it?
Re:Ridiculous (Score:3, Interesting)
Yes, I belive Edison tried over 5000 different hand made bulb/filiment combinations before he found one that supported his assertion.
Thowing out data is not about proving pet theories, it's about admitting you cocked up the experiment. eg: Prof Sumner Miller [abc.net.au] never edited out failed demonstrations from his TV show, nor did he claim the failed demo proved accepted theories of physics were wrong, rather he would simply exclaim - "Experiments never fail, it is I who have failed to set the right conditions for nature to cooperate" and then try again.
Re:Why most scientists and engineers screw up (Score:4, Interesting)
Or when the results you get aren't acceptable to the people responsible for continued funding.
Years ago, I worked for months trying to reproduce the Polywater research,
http://en.wikipedia.org/wiki/Polywater
and eventually reported that I was unable to do so.
The department considered my work a failure (as in, I must have been incompetent) and did not publish my findings. When, years later, the publications reporting successful discovery/creation of Polywater were shown to be fraudulent, and my results were correct, I did not even receive an apology.
Throwing out results is unethical as well as irresponsible. Many discoveries have come from re-evaluating what appears to be "bad" data. It might not be possible to use it now, but it should be at least stored.
For instance, it has been reported that the "bit of "scruff" on her chart-recorder papers that tracked across the sky with the stars"[1] looked like bad data to Jocelyn Bell Burnell's supervisors. Today we call the phenomenon a pulsar.
[1] Wikipedia
Re:Why most scientists and engineers screw up (Score:3, Interesting)
Really? Or is it that you are SO politically correct that you cannot see truth.
I happen to have mod points and my on-the-fly ranking went from insightful to interesting to troll and back to interesting.
I've lived long enough to understand that each of the 6 billion people on this earth is different than every other. Some are remarkably good and some are remarkably bad. Most of us are just average in our own interesting ways.
But still, I do believe that genetic differences affect what we are and that genetic differences can be attributed to where our genes came from.
Those who choose to never risk offending anyone are perhaps the most intellectually dishonest among us.
Should I post this or should I mod YOU the troll?
Exceedingly common (Score:2, Interesting)
in my experienced - I'm a physical chemist doing atomic resolution condensed phase computer modeling. It's so common that I am troubled when the first analysis gives the answer I expected. I likely spend more time looking for errors when the answer makes sense the first go through. Really.
Re:Ridiculous (Score:3, Interesting)
Very true. Sometimes the data really are wrong. (The Minimum Discrimination Information criterion [wikipedia.org] is a way of rigorously answering the question of whether your data or your theory is in error.)
But simply throwing out the data is the 100% wrong thing to do. That destroys the information that would have eventually told you if you were really doing something wrong in your experiment, or if you've discovered something new.
It also creates an information cascade-type situation where everyone culls any non-conforming data they see, and then all available data is conforming, which makes later scientists more skeptical of future non-conforming data, and so on. This cascade can make it so that even a randomly-chosen theory could be supported by the literature, even in the face of being completely unrelated to reality.
Supposedly, that's what happened with the Millikan oil drop experiment and everyone biasing themselves in the direction of his initial, wrong value for electron charge.
Re:Why most scientists and engineers screw up (Score:4, Interesting)
Here's the problem. If you can't order every single human into one race or another, your model is flawed. If you're forced to resort to mixes of races, well, then you don't have any distinct race left.
Race concepts fall apart once actual taxonomic principles are applied to them. Your examples actually illustrate the problem quite nicely: not nearly all asians have problems with milk - specifically the Japanese the do. Indians (from the Indian subcontinent in Asia) do not. Blacks do not have a higher tendency for sickle-cell anemia, a certain group of people in Africa do. Blacks in the US do not have that trait.
How much does it suck to be so wrong? Your cognitive dissonance must be at a record high.
Re:Why most scientists and engineers screw up (Score:3, Interesting)
Which is why people who claim scientists only care about the truth are wrong. You only cared about the truth and were fired. Plenty of other people would have been looking out for their job first, and made sure their results confirmed what the department expected.
That said- the great thing about science is that eventually the truth will be discovered despite the pressure for money/jobs. It may not happen in a lifetime, but as long as science continues, it will happen.