Bloggers Put Scientific Method To the Test 154
ananyo writes "Scrounging chemicals and equipment in their spare time, a team of chemistry bloggers is trying to replicate published protocols for making molecules. The researchers want to check how easy it is to repeat the recipes that scientists report in papers — and are inviting fellow chemists to join them. Blogger See Arr Oh, chemistry graduate student Matt Katcher from Princeton, New Jersey, and two bloggers called Organometallica and BRSM, have together launched Blog Syn, in which they report their progress online. Among the frustrations that led the team to set up Blog Syn are claims that reactions yield products in greater amounts than seems reasonable, and scanty detail about specific conditions in which to run reactions. In some cases, reactions are reported which seem too good to be true — such as a 2009 paper which was corrected within 24 hours by web-savvy chemists live-blogging the experiment; an episode which partially inspired Blog Syn. According to chemist Peter Scott of the University of Warwick in Coventry, UK, synthetic chemists spend most of their time getting published reactions to work. 'That is the elephant in the room of synthetic chemistry.'"
Terrible, Terrible, Headline (Score:5, Insightful)
Re: (Score:2, Interesting)
The bloggers are not testing the scientific method, they are testing methods that are scientific
Putting your ignorance in boldface type is amusing. The most basic promise of the scientific method is that results can be replicated by anyone with the proper equipment repeatedly and reliably. This is accomplished by describing an experiment to the detail level necessary to reproduce the result. If the result cannot be reproduced from a description of the experiment, it has failed this test.
They are testing the scientific method insofar as asking whether professional and peer-reviewed scientific work actu
Re:Terrible, Terrible, Headline (Score:5, Informative)
They are testing whether scientific papers meet the scientific method (ie. the results are reproducible). They are not testing the validity of the scientific method itself (myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question).
That is the point GP was attempting to make.
Re:Terrible, Terrible, Headline (Score:5, Funny)
myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question
So little faith you have...
(large grin)
How about unpublished protocols ? (Score:2)
There are protocols which are published and then there are protocols that remains unpublished
How about those protocols which have remained unpublished?
Any way to test those?
Re: (Score:2)
Re: (Score:2)
There are protocols which are published and then there are protocols that remains unpublished
How about those protocols which have remained unpublished?
Any way to test those?
That's not science. That's grandma's secret recipe.
Re: (Score:2)
That which cannot be patented or copyrighted remains secret by necessity. Lest someone else profit from your work, which cannot be permitted.
It's all about money. All of it. All the time. For everyone. Accept this, and you will do more than survive, perhaps. Ignore this, and you will suffer. Reject this, and you will fail, either in your philosophy or in your efforts. You don't have to like it nor do you have to perpetuate it, but it is what it is.
Re: (Score:2)
How about those protocols which have remained unpublished?
Ok... for you, let me be as explicit as possible: put you faith in God and pray!
Is it clearer now?
If not, do consider the context (or else [xkcd.com]).
Still confused? I hope this will clear the matter: whoosh!
Re: (Score:2)
How about those protocols which have remained unpublished?
You need to test "Just make it up so that results give the answer we want in order to get more funding"? Really?
Not that good scientists ever use that method. Alas, not all scientists are good, and that's why it is important to check published results. (It's a good task for grad students...)
Re: (Score:2)
Re: (Score:2)
Well, there are the unpublished protocols, and the published unpublished protocols, and finally the unpublished unpublished protocols.
(Just ask Rummy)
Re: (Score:1)
They are testing whether scientific papers meet the scientific method (ie. the results are reproducible). They are not testing the validity of the scientific method itself (myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question).
Epistemology [wikipedia.org] would have something to say about your sight. Yes, the scientific method can be evaluated and tested. Anyway, if that's what you say the GP is making as a point, then it's a really stupid and pedantic one. The scientific method isn't just an abstract concept, it's also a means to an end. If we find that many people are making the same mistakes in the process itself, then yes, it is a statement about the scientific method.
It's like saying a lake is a body of water and ignoring the fact that you
Re: (Score:2)
No, I have not studied philosophy with any serious effort, but if the scientific method cannot be used to prove itself, then perhaps it is lacking?
And I don't reject the 'scientific method'. I think it is self-evident and self-revealing.
The real question of these papers is not can the results be shown to be reproducible, but did the papers sufficiently disclose the methods? And if not, why not? there is, naturally, the possibility that the results were not even reproducible by the authors, and so they eit
Re: (Score:2)
No, I have not studied philosophy with any serious effort, but if the scientific method cannot be used to prove itself, then perhaps it is lacking?
It's more a matter of logic.
Assume, for the sake of argument, that the scientific method is flawed. Trying to test the scientific method could produce myriad results. Tests could show that it always fails. Tests could give random results. Or tests could correlate with some random thing - imagine if the scientific method only works with a waxing gibbous moon is ascendant in Leo. Or, and this case is significant, tests could always show that it does work (but it then fails when you try to test other theories)
Re: (Score:2)
Lines that intersect are not, by definition, parallel. Lines either intersect or they do not. Lines that do not intersect are parallel.
We define non-intersecting lines as parallel. We don't prove they are parallel, we describe them as such.
It has been a long time since I've employed geometry. How that relates to proving the Scientific Method is beyond me.
Re: (Score:2)
If 'Peer Reviewed' doesn't include replicating results (which would delay publication and we can't have THAT!), then it is reduced to 'nice penmanship'.
Which, of course, we know is true. Peer review is just self-ratification. No one else could possibly understand, much less reproduce, serious and groundbreaking work, so just step back and watch the show. Pay at the door.
Re: (Score:3)
This speaks to the failings of the participants implementation of the scientific method, not to the failing of The Scientific Method.
We have discussed here before the problem of too much content being generated and not enough people to peer review it all. Still not a failing of The Scientific Method.
And I prefer my ignorance in courier font.
Re:Terrible, Terrible, Headline (Score:4, Insightful)
Re: (Score:2)
Not describing the experiment correctly will not bother their corporate sponsors, who will either employ them directly or get the full description upon payment. Never doing it at all? No problem if they get more grants to build upon the nonexistent foundation they fabricated - unless they change direction with every grant and never evolve their previous work, which is nice work if you can get it.
Re: (Score:2)
Not describing the experiment correctly will not bother their corporate sponsors, who will either employ them directly or get the full description upon payment. Never doing it at all? No problem if they get more grants to build upon the nonexistent foundation they fabricated - unless they change direction with every grant and never evolve their previous work, which is nice work if you can get it.
Honestly, I think the most probable reason is poor record taking and poor writing.
Re: (Score:2)
Sadly, you are quite possibly correct.
So much for the scientific method. Even the experts suck at it.
Re: (Score:2)
Sadly, you are quite possibly correct.
So much for the scientific method. Even the experts suck at it.
Nobody said science was easy.
Re: (Score:3)
We have discussed here before the problem of too much content being generated and not enough people to peer review it all. Still not a failing of The Scientific Method.
The article's headline text is a demonstration of that issue of too much content and not enough peer review :)
Re:Terrible, Terrible, Headline (Score:4, Informative)
The bloggers are not testing the scientific method, they are testing methods that are scientific
Putting your ignorance in boldface type is amusing. The most basic promise of the scientific method is that results can be replicated by anyone with the proper equipment repeatedly and reliably
And I am sorry that you struggle so greatly to understand what I have written.
They are testing the scientific method insofar as asking whether professional and peer-reviewed scientific work actually meets this basic test.
Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.
Hell, if you go back and actually read my comment - I would say re-read but it does not appear you read it successfully for a first time yet - you will find that I did say this work is important. I also said that it is not testing the scientific method itself, which is correct.
Re: (Score:3)
Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.
Many of the published results and methods being verified are ones that have questionable results - such as producing too much output chemicals or reactions don't appear they should work at all. Those are papers for which peer-review has failed to provide adequate review. If the paper was truly read in-depth by other equally qualified scientists these issues would have been noticed and the paper (published or not) would have been called into question.
The caveat to this would be papers that are published w
Re: (Score:3)
Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.
Many of the published results and methods being verified are ones that have questionable results - such as producing too much output chemicals or reactions don't appear they should work at all. Those are papers for which peer-review has failed to provide adequate review. If the paper was truly read in-depth by other equally qualified scientists these issues would have been noticed and the paper (published or not) would have been called into question.
The caveat to this would be papers that are published with the sole purpose of seeking peer review and inviting other to validate the results, for example many of the cold fusion papers and the experiment which implied neutrinos were traveling faster than light.
I also recognize that peer review happens both before and after publication, and in fact the bloggers are part of the peer-review process.
Also frankly, there's a ton of papers out there which do work but which omit crucial details of how they work. Queue 2 years of my life discovering that someone's "simple robust synthesis" can't possibly have worked the way they said it did, and gradually reverse engineering that they were doing something they didn't report in any papers which had an important effect (they didn't quite seal up the reaction vial, but didn't leave it open, so coupled with water and condensation they were setting up a highly u
Re: (Score:2)
That's not testing the scientific method. Testing the scientific method would involve some test as to whether the scientific method itself works. Determining if some published experiment is actually described in a way that is reproducible says nothing about whether the scientific method itself works or is useful.
Asking whether professional and peer-reviewed scientific work is actually using the scientific method is also not asking whether the scientific method itself works or is useful.
I agree it's a useful
Re: (Score:2)
they are testing methods that are scientific.
If a lot of those experiments can't be reproduced, then those methods weren't scientific to begin with.
Re:Terrible, Terrible, Headline (Score:5, Insightful)
Right, but they are utilizing the scientific method to test the quality of published papers, not attempting to verify the utility of the scientific method itself.
The headline should read "Bloggers apply scientific method to validate published findings".
Re:Terrible, Terrible, Headline (Score:5, Insightful)
"Bloggers apply scientific method to validate published findings".
A much better headline.
Re: (Score:2)
...or alternately: The Half Blood Prince tweaks the results.
Re: (Score:2)
Re: (Score:2)
Could as well say "Bloggers do what tens of thousands of grad students do every day."
That's about the face of it. The only difference is that they're blogging their progress rather than simply trying to get an intermediate step to work because you wanted to make something that is on the route to what you're really studying and you just want a decent prep for it.
I think the general rule of thumb is "subtract at least 15% from the expected yield and assume at least one crucial step has been inadequately documented" and "ignore esoteric and exotic purification method and just recrystalise it".
Re: (Score:3)
Then the scientific method is effectively shot, not because it's invalid, but because it's not being followed, and it becomes no longer reasonable to believe researchers are following it
That is identical to:
People who don't do up their seatbelt buckles die.
Therefore, seat belts fail to protect people.
The fact that people are failing to apply the scientific method does not mean that the scientific method has failed, only that some people who call themselves "scientists" shouldn't.
Re: (Score:2)
If a large proportion of researchers claiming to use the scientific method either can't or won't actually use it, the method has failed in some way. How is saying it would work if only human nature changed any different from making the same claim about anything else, say Communism? How do you prove the method itself is correct if most of your test subjects actually fail to understand how to use it, yet in this case, your test subjects are supposedly all well trained expert users? How do you prove the method
Re: (Score:2)
People who don't do up their seatbelt buckles die.
Therefore, seat belts fail to protect people.
If researchers show people on average say they do up their seat belts but are actually fibbing, lying, or mistaken, and don't actually do up their seatbelts properly, then, yes, it does show seat belts fail to protect people.
But more importantly: it says, you can't trust people when they say they do up a seatbealt.
Which is where you and your analogy fall flat on the face.
It doesn't physically effect othe
Re: (Score:2)
they are testing methods that are scientific.
If a lot of those experiments can't be reproduced, then those methods weren't scientific to begin with.
Not necessarily. Sure, irreproducible results can be the result of shoddy, missing, or fabricated work, and unfortunately often are. There are also times when those results can come about through no fault the experimenter. If the scientist was running an overnight synthesis and was not notified that the HVAC failed for two hours starting at 3am is that his fault? Sure, he should check afterwards to make sure that the environmental conditions were properly controlled but that isn't always immediately ap
Re: (Score:3)
A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results. If you can't repeat your results you can't possibly give others instructions on how they can repeat them. Not knowing that the HVAC failed for a couple hours during one run out of a dozen should result in outlier results that can be investigated or discarded.
Re: (Score:2)
A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results. If you can't repeat your results you can't possibly give others instructions on how they can repeat them. Not knowing that the HVAC failed for a couple hours during one run out of a dozen should result in outlier results that can be investigated or discarded.
Unfortunately, sometimes the experiments do get run multiple times and the data that didn't fit the expected results was thrown out and not included in the final data. I've seen far too many scientists who automatically throw out the highest and lowest samples and only average the data that grouped nicely, without making much effort to find out why some samples deviated.
Re: (Score:2)
A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results.
No time. Many if not most synthetic methodology papers will test a new reaction on a range of homologous compounds, say 20. Few of those are repeated twice, let alone multiple times. A lot of this work comes out of graduate school groups where the emphasis is on publishing rapidly. There is one Nobel Prize winner in particular whose publications in Tetrahedron Letters (usually a two page paper) are notorious for sacrificing accuracy for rapidity.
Re: (Score:2)
Re: (Score:2)
I thought Tetrahedron Letters in general was notorious for allowing very low quality articles to be printed. There must be a reason my professor in organic chemistry insisted on calling it "tetrahedron liars".
TettLet is for rapid publications. The methods are not always fully worked out and there is no detailed experimental. The function of TetLett is to give the chemical world a heads up of a new way of doing something. Where I expect a paper from the Journal of Organic Chemistry to be detailed, I don't expect the same of TettLet.
Re:Terrible, Terrible, Headline (Score:5, Insightful)
A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results.
Won't help. I studied to be a chemist (admittedly a long time ago) and by far the biggest non-ethical problem out there is contamination.
So it turns out that your peculiar reaction you're studying is iron catalyzed, in fact its incredibly sensitive to iron, but no one in the world knows that yet. And your reagents are contaminated. Or your glassware, which you thought was brand new and/or well cleaned, is contaminated. Or your lab is downwind of a hematite ore processor and the room dust is contaminated.
Sure, you say, test everything. Well there isn't time/money for that, but for the sake of argument we'll assume there is. What if dust from the hematite ore processor is far larger than the filter paper pores in filtration stage of your overall process? Test the reagents and product all you want but you'll never find iron anywhere except room dust (which you already knew about) and the debris in the filter paper (which you assume was contaminated by room dust AFTER removal from the apparatus)
The most important thing is this is the norm in chemistry, not an outlier. Chemistry is not math or CS, sometimes stuff just doesn't work or just works for no apparent reason. Unlike some technologies, detailed modeling of "why" "how" often doesn't happen for years, decades, centuries after the ChemEng team has been selling product / papers have been written.
A very important lesson is analysis paralysis. So you live downwind of a hematite ore crusher. And you know it. And periodic tests of your lab show iron enriched house dust. But you can't go around testing everything, because you're surrounded by millions of things to test for. You're a gardener, god only knows whats on your hands. Skin oil of certain blood types is a contaminant? Your breath has a tinge of ethanol in it from last night? Maybe its your perfume / cologne / antiperspirant / nail polish? The point of discovery is its literally unknown... maybe wearing nitrile gloves instead of old fashioned latex "does something" good or bad to the reaction.
I think the main thing "slashdotter IT people" need to understand is most chemistry and most chemical engineering runs somewhat less than 6 sig figs. This is incomprehensible to IT people... if your T1 or whatever LAN had a BER worse than 1e5 you'd call it in for repair... If you got one thousand read errors when you read a 1 gig DVD, you'd throw out that DVD. If your processor runs at 1500 mips then at a six sig fig error rate it would crash about 1500 times per second. The bad news is six sig figs is actually pretty good work for a chemistry lab. Certainly undergrads could never aspire it that level, both skill and experience and specialized equipment....
Please spare me the details of one peculiar quantitative analysis technique that in one weird anecdote measured once in tiny fractions of a ppt. The overall system cannot be "cleaner" than the filthiest link in the entire system. Is the hand soap in your lab spectrographically pure? The unused toilet paper in the bathroom? All of your hoods and benches and storage cabinets are in a verified and tested cleanroom environment? Seriously? The drinking cooler and lab fridge also only hold spectrographically pure substances? Please no anecdotes.
Re: (Score:1)
Oh snap!
Re: (Score:2)
Re: (Score:2)
Right, I tagged the story "crapheadline" as soon as I RTFS.
It might be epic (Score:5, Interesting)
The bloggers are not testing the scientific method, they are testing methods that are scientific. Those are two vastly different concepts. Their work is important, but not epic.
I'm not so sure about that.
We believe in a scientific method founded on observation and reproducible results, but for a great number of papers the results are not reproduced.
Taking soft sciences into consideration (psychology, social sciences, medical), most papers hinge on a 95% confidence level [explainxkcd.com]. This means that 1 out of every 20 results arise from chance, and no one bothers to check.
Recent reports tell us depression meds are no better than chance [www.cwcc.us] and scientists can only replicate 11% of cancer studies [nbcnews.com], so perhaps the ratio is higher than 1 in 20. And no one bothers to check.
I've read many follow-on studies in behavioral psychology where the researchers didn't bother to check the original results, and it all seems 'kinda fishy to me. Perhaps wide swaths of behavioral psychology have no foundation; or not, we can't really tell because the studies haven't been reproduced.
And finally, each of us has an "ontology" (ie - a representation of knowledge) which is used to convey information. If I tell you a recipe, I'm actually calling out bits of your ontology by name: add 3 cups of flour, mix, bake at 400 degrees, &c.
This assumes that your ontology is the same as mine, or similar enough that the differences are not relevant. If I say "mix", I assume that your mental image of "mix" is the same as mine. ...but people screw up recipes, don't understand assembly instructions, and are confused by small nuanced differences in documentation.
Does this happen in chemistry?
(Ignoring the view that reactions can depend on aspects that the researchers were unaware of, or didn't think were relevant. One researcher told me that one of her assistants could always make the reaction work but no one else could. Turns out that the assistant didn't rinse the glassware very well after washing, leaving behind a tiny bit of soap.)
It's good that people are reproducing studies. Undergrads and post-grads should reproduce results as part of their training, and successful attempts should be published - if only as a footnote to the original paper ("this result was reproduced by the following 5 teams..."). It's good practice for them, it will hold the original research to a higher standard, and eliminate the 1 out of 20 irreproducible results.
Also, reproducing the results might add insight into descriptive weaknesses, and might inform better descriptions. Perhaps results should be kept "Wikipedia" style, where people can annotate and comment on the descriptions for better clarity.
But then again, that's a lot of work. What was the goal, again?
Re: (Score:3)
I have mod points, but stupid Slashdot isn't showing me the moderation option on posts. So I will snark instead.
But then again, that's a lot of work. What was the goal, again?
Uhm. Publish or perish, I think it was...... Full speed ahead and damn the torpedoes. Verification studies don't count on your CV.
Good idea to harness the slave labor though. That's what grad students are for.
And how many grad students will actually be willing to do this verification work? None, who can think politically enough to stay in academia. What are you asking them to do? Verify pu
Re:It might be epic (Score:4, Interesting)
Many commenters seem to be mistaking some idealised thing called the Scientific Method with what is actually practiced in the real world when they claim that the scientific method is not being tested. Damn straight this is testing the scientific method - warts and all. If the scientific method as practiced in the real world does not deliver papers that provide sufficient detail to be reproduced then it is not working properly and is broken. In fact, I'm sure that most people who actually publish papers will acknowledge that the peer review process is broken and does not approach this idealised notion of what peer review is in many people's minds. If peer review is broken then a critical element of the scientific method is also broken.
Reproducing results is hard every time I have done it - and that includes times when I have had access to the exact data used by the authors. (And sometimes even the exact code used by the authors - because I was using a later version of the statistical package and results were not consistent between versions.)
So, if people want to claim that the Scientific Method is perfect and this is not a test of it - it would be interesting if they could point to anywhere this idealised Scientific Method is actually practiced (as opposed the the flawed implementation that seems to be practiced in pretty much any field I have ever become acquainted with).
Re: (Score:2)
And finally, each of us has an "ontology" (ie - a representation of knowledge) which is used to convey information. If I tell you a recipe, I'm actually calling out bits of your ontology by name: add 3 cups of flour, mix, bake at 400 degrees, &c.
This assumes that your ontology is the same as mine, or similar enough that the differences are not relevant. If I say "mix", I assume that your mental image of "mix" is the same as mine. ...but people screw up recipes, don't understand assembly instructions, and are confused by small nuanced differences in documentation.
Does this happen in chemistry?
It absolutely happens - chemistry is like cooking in that respect. Especially some of the trickier synthetic chemistry where your technique really matters. Chemists of different skill levels are going to get vastly different results from following the same prep.
What these guys are doing is no different to what grad students do all the time - papers are a great source of preps for compounds that you want to use as building blocks for other things you want to do if your starting materials are either not readi
Re: (Score:3)
It's good that people are reproducing studies. Undergrads and post-grads should reproduce results as part of their training, and successful attempts should be published
Dang, you were doing so well too. How about we publish both the successes and the failures, rather than just the successes. Just publishing the successes is how we get stuff like the 11% reproducibility rates you talked about earlier. So a footnote in the original paper: "this procedure was replicated by 23 teams, those results supported the results of this paper in only 3 cases."
Re: (Score:2)
Since testing IS the scientific method, if there is something wrong with the scientific method, this test will fail!
Not just synth chemistry (Score:3, Informative)
If you try to repeat an experiment and fail then it is almost impossible to get published. Failed experiments, though critical for advancing science, aren't sexy and editors prefer their journals to be full of positives. So scientists don't even bother trying anymore. This is a problem in medicine and probably all sciences. There is a movement in medicine to report all trials [alltrials.net] so they can be found by researchers doing meta-studies.
Re: (Score:2)
Tell that to Fleischmann and Pons.
Re:Not just synth chemistry (Score:4, Interesting)
Re: (Score:2)
Watching scientists work should be like hearing one guy say, "Hey, check this out!" and another guy saying, "wow, that's cool, I can't get it." and another saying, "oh, you need to do it like this!"
If someone can't say, "that doesn't work like you said it did," then scientific progress is going to be hampered.
Re: (Score:1)
That doesn't sound like a very good system.
It's one that works, and is another part of the scientific method, albeit a step along from establishing reproducibility of an experiment. If prior research appears to set a hypothesis, then that should be tested - i.e. someone should make an attempt to knock it down by whatever means are available. You never really "prove" something to be the case, but if nothing can be done to disprove it, that's pretty nearly as good.
Re: (Score:3)
Re: (Score:2)
It's not a problem with the scientific method, it's a problem of communicating results. Clearly it isn't working optimally if this article is correct. If people can only tell each other when something works, but can't discuss when things don't work, then the communication channels are severely broken.
This is actually a huge problem.
Science really needs something like Freenode for different disciplines. A lot of the time you simply want to be able to talk to some of the people dealing with the work in the original group, since they're likely to be able to give you the small details which turn out to matter.
Re: (Score:2)
That doesn't sound like a very good system.
Watching scientists work should be like hearing one guy say, "Hey, check this out!" and another guy saying, "wow, that's cool, I can't get it." and another saying, "oh, you need to do it like this!"
If someone can't say, "that doesn't work like you said it did," then scientific progress is going to be hampered.
It's how it works though. Usually it's "hey look at this, and this happens because of x reason" and you come back with "I don't think that works the way you think because of y systematic error or z error in your technique". I've personally done this before with published papers, and it doesn't necessarily invalidate them so much as add more information to the table.
Mythbusters: Chem Edition (Score:2)
Seriously? IDIOTS (Score:1)
Could this be why? (Score:2)
Could this be why the general public doesn't trust science and instead rely on ancient mystical texts to make sense of the world they live in? Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?
Re: (Score:2)
Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?
How many of the "hoi poloi" would have any idea what the paper is even trying to demonstrate? Or how to test it?
Re: (Score:2)
Could this be why the general public doesn't trust science and instead rely on ancient mystical texts to make sense of the world they live in? Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?
Not really. Also, it wouldn't matter - being a skilled synthetic chemist is a lot more than simply following the instructions, no matter how well they are written out.
Perhaps you could make it easy enough for an undergrad with some reasonable lab experience to do it, but the problem with public perception of science is not optimistic yield reports and poor experimental write up in research papers.
Re: (Score:2)
I can tell you're the real article. It's well known that physicists have no concept of paragraph breaks. ;)
More seriously, thank you for the excellent post. Which, BTW, backs up everything that my dad (retired physics/math prof) has been complaining about in this area for the last 20 years or so.
Re:Seriously? IDIOTS (Score:4, Insightful)
Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down.
Organic chemistry is quite difficult. The purpose of synthesis is not as you suppose, just mix A and B, see what happens and publish. Most organic chemists are trying to make specific transformations on certain parts of molecules in high conversion and trying to control the variables of time, temperature, concentration, reagent reactivity with substrate functional groups, etc.
Physics is just a block on an inclined plane and variations.
Re: (Score:2)
Physics is just a block on an inclined plane and variations.
Stop talking shite. You obviously don't know what you're talking about, all disciplines of science are difficult and complex. If I so wished I could be just as dismissive and say chemistry is just dirty, poorly explained physics (as some physicists up their own arses think). The boundary between physics, material science, and chemistry is very blurred. We use many of the same techniques, just with different approaches, language, and mathematical models. What the chemist is interested in the physicist doesn'
Re: (Score:2)
Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down. Their papers are almost universally set forth as an over glorified recipe. If you look across the fields chem majors have a higher percentage of publishing as undergrads, because the papers are so damn easy. You mix your bits, filter and then quantify the results. Boom, paper. Yes I'm oversimplifying, but not by much. The joke goes that if you want to make a breakthrough in modern chemistry you'd better find a physicist. If you want a breakthrough in modern physics you'd better find a mathematician. The overarching problem is that the system pushes for papers period. Not worthwhile science, not correct science, simply papers.
By your logic Physics is even easier that Chemistry; just push "go" in LabView and publish the resulting graph. And Biology practically researches itself; just shake up a jar of bacteria and watch them grow.
The nuances of synthetic chemistry--a sub-discipline of Chemistry--are far more complex than mixing A and B, which is the core problem of reproducibility. If you can prove the structure, then you obviously made it, but conveying precisely what you did can be a challenge as there are too many factors to c
Re: (Score:2)
Chemistry is among the easiest of the physical sciences. I say this as the physicist
That just created the image in my mind from an episode of The Big Bang Theory of Dr. Sheldon Cooper at the inter-departmental paintball tournament shouting "Geology isn't a real science!"
Re: (Score:2)
Replicating chemistry isn't hard with good instructions.
Organic chemists often lump reactions into two classes: peak reactions and plateau reactions. Peak reactions give a maximum yield of product only for a narrow range of conditions. Plateau reactions work over a broad range of conditions. If you go to run a plateau reaction, you don't need a very good SOP, but if you are running a peak reaction you need very specific conditions and detailed instructions for every step. And you generally don't know what type of reaction you are going to have on your hands just
Re: (Score:2)
If this distinction is so important and difficult to figure out, why is it almost never mentioned which class of reaction is going on in the paper that describes the synthesis? Why don't reviewers, and other chemists, demand that such information be included?
Sometimes the researchers will mention it when discussing their results. Plateau reactions are good, and if you find one you tout it, but often the original researcher has not run enough variations on the conditions to even be able to define it. It is often later when several papers are available that you realize that the new reagent, say, works well hot or cold, works well in protic or aprotic solvents, tolerates a wide range of functional groups, etc.
Papers are not little textbooks. We don't expect th
Re: (Score:2)
I'm just saying these people sometimes get away with doing an extremely poor job at documentation.
They're not the only ones... it happens in IT a whole lot, as well.
Good luck with that (Score:5, Insightful)
Re: (Score:3)
It's not a secret that about half of published synthesis methods are garbage and yield values are wildly creative. Reviewers don't have the means to verify these, so anything that seems plausible gets published. Then researchers are left to sort out the best methods based on which ones get the most citations.
It's not just synthesis methods. I remember taking a graduate control theory class in which the final project was for the class to replicate the results of a paper with a particular control algorithm. It just...wouldn't...work. Not a single person managed to replicate the results, which simply led to the inevitable conclusion they were fudged.
I'm not and would never defend anyone who publishes any data that has been tempered with, but I still find it annoying that we've set up a system where there's an i
Its called... (Score:2)
calling out BS, exaggerated, wrongly calibrated, and/or embellished results. This makes perfect sense to me. If you publishing a paper on a subject then it should be a repeatable recipe.
Re: (Score:2)
And yet, I would argue it's an ongoing disaster that a lot of the time we're not really trying to quantify these types of parameters. Nothing is gained if we require everyone to go through the exact same learning exercise, and we never actually dissect the common problems and figure out what causes them.
Which scientific method are they testing? (Score:2)
Are they testing the tried and true scientific method that *real* scientists used for centuries to arrive at the cumulative knowledge of mankind, or are they testing the modern scientific method that involves drawing a conclusion and then trying to find data that fits your model, discarding any data that doesn't?
Chemistry (Score:2, Funny)
It's the major people that can't handle physics switch to.
(I kid, I kid)
Re: (Score:2)
It's the major people that can't handle physics switch to.
Chemistry is just applied physics. [xkcd.com]
Combine Forces (Score:2)
The Open Science Framework focuses on psychology instead of chemistry but I would imagine they could both utilize similar frameworks and methodologies. I'm not a chemist or a psychologist so there may be some major incompatibility I don't know about.
Re: (Score:2)
A consequence of publish or perish (Score:1)
I suppose reviewers are supposed to catch these things but they're probably too busy with their own 'discoveries' to give more than a cursory glance. Hopefully the bloggers will get enough recognition from their efforts to spur others; especially if the trend towards publishing direct online and away from peer reviewed journals continues.
good, but... (Score:2)
Replicating published experiments is a worthwhile effort, and there should be more of it. However, it's already pretty well known that scientific papers have a high rate of irreproducible results, due to both fraud and error. If they fail to replicate particular experiments, it's not an indictment of the scientific method, but of the particular scientists.
(Also, some experiments in the natural sciences are tricky to do and require experience to work, but they can still be replicated.)
The work of Harold Garfinkel (Score:3)
is instructive (though still not very readable) here.
The founding ethnomethodologist, Garfinkel argued that much of science depends on practical assumptions and habits of which researchers are only vaguely aware, leading to the "loss" of the phenomenon.
This is both good and bad. On the one hand, it means that a phenomenon is real, with real implications (useless theory and tautology are marked by the difficulty of losing the phenomenon), but on the other hand it means that what is said about the phenomenon is often missing the most critical bits of information, unbeknownst to the PIs themselves because they are unaware of practical (embedded in the practice of) assumptions and habits, something that makes it seem likely that many scientific truths are either solidified far later than they might otherwise have been or incorrectly lost to falsification rather than pursued.
Things I Won't Work With (Score:2)
It's by Derek Lowe, a pharmaceutical chemist who blogs about chemical compounds that are way too dangerous. His position is that the closest you want to get to any of these things is reading about them. The closest I want to get is reading what he has to say about them.
Take FOOF, aka F2O2, aka Dioxygen Difloride. Lowe calls it "Satan's kimchi".
Re: (Score:2)
Thanks for that. That's hilarious.
A very old elephant in the room (Score:3)
These many many decades ago I went to a "top 5 STEM school" and enrolled in Chemistry. While all of the students had Very Expensive Machines to play with, they also had No Time To Do Careful Work.
"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.
People like to harsh on the 'soft' subjects like history ... but I can assure you that many a history teacher would chew you up one side and down the other for saying things that are patently false and blatantly contradict the evidence - while many a Chemistry teacher would simply tell you "well, it should have worked. and you understand the basic ideas. so lets move on."
I'm not sure what the issue is - if it's just too expensive to do experiments, or maybe the point is not to learn 'experimenting' but rather to learn 'theories... with some hands on experimenting to give you a flavor of it"????
Re: (Score:2)
"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.
See... research teams should have separation of duties, to avoid the temptation to cheat. The scientist does the experiment, someone else takes the measurements, someone else records the results, and enters into digital format; which quickly become immutable, digitally signed and timestamped; with noone taking or recording measurements allowed t
Re: (Score:2)
The e
Re: (Score:2)
Did he bust you for error bars / sig figs? How many sigma was your "signal" above the noise? "pointed toward" doesn't sound like a firm result, so I can see the old prof getting kinda wound up if you wrote a relatively strong conclusion in English language prose, but provided math showing a very weak conclusion. That's "TV commercial logic" not scientific logic. Still the guy was a bastard about it, because he didn't explain his (likely) reasoning to you, or at least he didn't do it very well.
Narrowing
Re: (Score:2)
And looking back, the methodology wasn't great -- tube physics do funky things to sound, so measuring through the PVC wasn't a good thing; the 1 meter leng
Re: (Score:2)
These many many decades ago I went to a "top 5 STEM school" and enrolled in Chemistry. While all of the students had Very Expensive Machines to play with, they also had No Time To Do Careful Work.
"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.
People like to harsh on the 'soft' subjects like history ... but I can assure you that many a history teacher would chew you up one side and down the other for saying things that are patently false and blatantly contradict the evidence - while many a Chemistry teacher would simply tell you "well, it should have worked. and you understand the basic ideas. so lets move on."
I'm not sure what the issue is - if it's just too expensive to do experiments, or maybe the point is not to learn 'experimenting' but rather to learn 'theories... with some hands on experimenting to give you a flavor of it"????
Don't conflate education and research. If an experiment goes wrong in a classroom setting, then there simply isn't time/resources/patience to repeat it. For instance, when a student uses the wrong solvent and starts a fire, you grab the extinguisher, deal with the fire, and move on. Sometimes it is the students fault, sometimes the professor, other times the TA. Some people are just not meant to do chemistry--they are afraid of chemicals and/or they just don't "get it." Other people are just unlucky. It sim
computer science too (Score:3, Interesting)
The sad truth is that this happens everywhere, also in computer science. I once implemented a software pipelining algorithm based on a scientific paper, but when implemented, the algorithm appeared to be broken and had to be fixed first. In this case, it probably wasn't malice either. But the requirements to get something published are different from getting it to actually run. Brevity and readability may get in the way of correctness, which may easily go unnoted.
Re: (Score:2)
So true! When will CS conferences will require an open source demonstrator?
Some of the database conferences (SIGMOD, VLDB) have a "reproducibility" program. You can submit your code and results to a volunteer review team and get a "reproducibility stamp" with your paper if they can duplicate your results.
Not reproducible (Score:3, Insightful)
Me! Me! Me!... (Score:2)
Oooh, oooh, oooh I get to synthesize the MDMA!!! Rave anyone?
Re: (Score:3)
Like how in the 70s scientists all over the world proved and knew nuclear energy was causing an ice age but of course we all know how good their factual evidence turned out to be.
Or like how you're wrong about that, since there was no scientific consensus in the 1970s that Earth was headed into an ice age:
T. C. Peterson, W. M. Connolley, and J. Feck, "The myth of the 1970s global cooling scientific consensus", Bulletin of the American Meteorological Society, 89: 1325-1337, 2008.
Re: (Score:2)
I've been watching this project for awhile, and it seems to function a lot more like a flash mob than a live-blog.