Forgot your password?
typodupeerror
Science

Crowdsourcing and Scientific Truth 62

Posted by samzenpus
from the truth-by-committee dept.
ygslash writes "In an opinion piece in the New York Times Sunday Review, Jack Hitt states that comments posted to on-line articles, and elsewhere on line, have de facto become an important factor in what is accepted as scientific truth. From the article: 'Any article, journalistic or scientific, that sparks a debate typically winds up looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago. The truth is that every decent article now aspires to become the wiki of its own headline.'"
This discussion has been archived. No new comments can be posted.

Crowdsourcing and Scientific Truth

Comments Filter:
  • It's long been true that a top reason to go to academic conferences isn't only for the paper presentations, but rather for the hallway/dinner/bar conversations about those papers. More formally, many scientific journals will publish short letters or commentaries about papers they've previously published, and that practice used to be even more widespread (at some journals a "letter" has morphed into a mini-paper, but they used to really be letters to the editor).

    The same is now true online with something like Terence Tao's blog [wordpress.com]: it's interesting as much for what other mathematicians post in reply, as for what Tao himself posts (though his posts are quite interesting). The main difference as I see it is that the number of people participating is much greater (which has good and bad parts), and, in comparison to hallway conversations, the conversations persist and get referenced back to more, since they're in a semi-durable written medium (that's the "wiki-like" aspect the article discusses).

    • Some examples from my intellectual neck of the woods-- the comments sections were particularly interesting during the whole OPERA snafu, though with Lubos' blog in particular you have to deal with some pretty half-baked political ideologies.

      http://golem.ph.utexas.edu/~distler/blog/ [utexas.edu]
      http://motls.blogspot.com/ [blogspot.com]

    • by l3v1 (787564)
      It's long been true that a top reason to go to academic conferences isn't only for the paper presentations, but rather for the hallway/dinner/bar conversations about those papers.

      There is nothing similar between researchers talking with each other at a conference, and average commenters posting comments on a blog. Some will hate me for this but I have to say this is the same thing as comparing journalism with blogging (oh my, how many long and idiotic quarrels about this are out there). It is absolutely
    • by Pope (17780)

      Great link! Though I fear I've been on Slashdot too long: I misread his name as Terence Taco...

  • looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago

    What does this even mean? I can't parse it and its early Sunday morning but I'm not that drowsy or stupid (although some would disagree with the latter).

    Obviously its relevant that he references GOOD 700 year old vs just "a" 10 year old. Obviously its important that magazines are time filling fluff for the masses / chewing gum for the mind, and in the old days manuscripts held real individual contributions of science work (like a modern journal / preprint archive / e-journal). Manuscripts didn't do much

    • by Anonymous Coward

      He is referring to marginalia, or the practice of a scribe adding his comments and thoughts into the margins of a manuscript as he was copying it out. These notes would then be copied by the next scribe, how might add notes of his own. This could get rather extensive and later, during the high medieval period, it gave rise to the idea of a full glossary.

      • by vlm (69642)

        So how is this different than the modern magazine 10 years ago? You take a newswire story, add some local color, someones blog mentions it, etc.

      • by DingerX (847589) on Sunday May 06, 2012 @02:13PM (#39909741) Journal
        In truth, marginalia practicaly never make it into proper glosses. Glosses are usually assembled from authoritative texts that discuss the passage in question. And very few texts get the Gloss treatment: in the Latin world, it's the Bible, Corpus Iuris Civilis and Decretals above all. Some other texts might get glosses, but they rarely get a glossa ordinaria-class treatment.

        And to the midrashim comment in TFA, I'd point out that Rashi did a bang-up job himself in Hebrew.

        For the scholastic Middle Ages, criticism usually took the form of "one doctor says this ..., for these reasons. I disagree, rather saying this, for these reasons. To his reasons, I reply..."

        Same as it ever was.
    • "What does this even mean?"

      Hard to tell for several reasons... among them that New York Times website is broken. To would-be OPs: please don't link to articles there. When I click on a link it tells me to log in, which is fine, but then it only gives me front page, never the article referenced by the link. And then, once logged in, if I click the link again (thinking it might take me there), it tells me to log in again even though I am already logged in!. Major, and I really mean major, web fail.

      But aside from that, it still doesn

  • 'Any article, journalistic or scientific, that sparks a debate typically winds up looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago.

    Not here. Two or three lines of summary are usually enough for the equivalent in comments of a thermonuclear war.

  • Astroturfing (Score:4, Informative)

    by PPH (736903) on Sunday May 06, 2012 @10:42AM (#39908217)

    Its interesting to observe how much commentary on articles is devoted to shouting down the opposition rather than making observations and/or corrections of the original content. Given the accelerated pace of such discussions on-line, the utility of spurious research in support of questionable legislation has been reduced significantly. In other words, if you spot an ivory-billed woodpecker today, your claim might not survive long enough to secure funding or implement conservation measures. That is; without your supporters declaring that the time for further research and comments is over and now its time to act.

    Today's crowdsourcing serves to reduce the half-life of bad science.

    • Ultimately the refutation of bad science depends on data, not blog entries.

      The arsenic based DNA article published in Science was not finally laid to rest by blog entries, but by a careful analysis of the DNA.

      • by PPH (736903)

        "Ultimately" being the key word here. Its true, you have to do the science properly to refute bad research and publication. But some of that bad research facilitates bad legislation or case law. And in these cases, its critical to call BS fast enough to stop such poorly conceived responses. Once bad law is in place, its supporters don't give a damn about its basis anymore. Often, the fight to rescind it is never undertaken.

        From TFA:

        Alex Sanders, who as a member of South Carolina's House of Representatives fought to preserve the land, told me that when people ask him where the ivory-bill is, he says, "I don't know where he is now, but I know where he was when we needed him."

        Do you think they abolished that national park they created when the woodpe

        • What makes you think that objections raised in that manner are going to be any more accurate than the original work?

          If legislatures are incompetent enough not to fix their errors this process can cause harm in either direction; through action or inaction.

          • by PPH (736903)

            What makes you think that objections raised in that manner are going to be any more accurate than the original work?

            Because, given time, science tends to progress from a state of less knowledge to one of more knowledge. Otherwise we'd be losing the wheel, fire, and TV dinners.

  • by Gimbal (2474818) on Sunday May 06, 2012 @10:47AM (#39908239)

    There is a huge difference between scientific truth and, alternately, popular perception. I don't even want to try to explain that, it's so obvious - and there may some be more pertinent matters to address, in this.

    I think we can accept that comments sections do not make much of a forum for development of scientific anything. Comments are comments. Comments are not journal articles. Comments can be said to be peer reviewed, to some extent, but then again, comments are not journal articles, comments need not follow any specific format for reporting of questions and results, comments are just comments.

    I'm afraid that that all may be beside any points raised in the linked article, however. What the article looks like to me, in all my sense of bias: It looks like a way of trying to excuse a lack of significant content in articles, in lieu of some kind of perceptual bias about comments. It think it's just as well for the birds, though I know it's been said, "It's the thought that counts."

    • by nashv (1479253) on Sunday May 06, 2012 @11:11AM (#39908409) Homepage

      Comments are comments. Comments are not journal articles. Comments can be said to be peer reviewed, to some extent, but then again, comments are not journal articles, comments need not follow any specific format for reporting of questions and results, comments are just comments.

      I did not RTFA. I second your point. But even if we were to take a more generous view of commenting sections, the problem of noise filtering remains. Comment sections are a perfect example of what Asimov said best :

      “Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'”

      The amount of effort required to parse through comments to find gems of significant value is enormous. I know that this is that age of the crowd and so on, but there are certain issues on which the opinion of the crowd has on average very little value because of the complexity of the topic and the years of experience required to make informed conclusions. The trade-off between expert opinion and open crowdsourcing varies widely depending on what is the topic under discussion, and the userbase of the particular site. Vaccines and autism on a Californian site, for example.

    • Don't worry, it's an article-writer noticing that the internet is doing something (crowd-sourcing research, which it did in this example), and trying to take credit for himself, in his own comment section.

      This article is the soul-searching of an author, trying to find relevance in the post-newspaper world.
    • by khallow (566160)

      I think we can accept that comments sections do not make much of a forum for development of scientific anything.

      So what? You don't need much of a forum for "development of scientific anything". There's a means to publish scientific knowledge via arXiv and a means to discuss it via blogs and community discussion sites. This argument boils down to "but they don't have formal peer review". Put that in and journals become obsolete.

  • Those who disagree (Score:4, Insightful)

    by ygslash (893445) on Sunday May 06, 2012 @11:45AM (#39908709) Journal

    As OP (here on /., not the author of the article), I'd like to re-raise here in the comments a point in my original post that got edited out:

    There are many who disagree with the thesis of TFA. It is interesting to note that they are trying to make their point - where else? - in the comments on the article, in comments here on ./, and elsewhere in the blogosphere.

  • Since "science" cannot prove historical events, the only thing left is opinion. By definition, if something is repeatable or testable, it cannot be "proven" by scientific methods. All you are left with is belief.
    • by Alex Belits (437) *

      Of course, it can. If there are ruins of a city somewhere, it is very much scientific conclusion that city was at some point founded and at some point destroyed. One does not have to build and destroy a city to "reproduce" it -- it's sufficient that anyone can look at the same ruins.

      Now, a question if the rulers of the city were insufferable assholes, mostly result in knowledge of opinions that are in no way scientific, even though most likely answer is yes.

    • by jc42 (318812) on Sunday May 06, 2012 @01:41PM (#39909569) Homepage Journal

      Since "science" cannot prove historical events, the only thing left is opinion. By definition, if something is repeatable or testable, it cannot be "proven" by scientific methods. All you are left with is belief.

      It'd be difficult to find a more misleading characterization of science.

      First off, as various historians and theoreticians of science have observed, scientific methods rarely (if ever) "prove" anything. Scientific methods are all based on 1) proposing explanations for observations, and the 2) attempting to disprove those explanations. After sufficiently many such attempts at disproof have failed, an explanation gets promoted to "hypothesis", and then to "theory". But these are only tentative, with further attempts at disproval continuing whenever anyone can come up with a new test that hasn't been tried.

      As part of this, an explanation that is untestable isn't considered scientific at all. It's neither true nor false by scientific standards, until someone comes up with tests that could possibly disprove it. Some explanations (e.g., "God did it") have remained in this state for centuries.

      Actually, there is one situation where there is a sort of scientific "proof". This is dealing with negative claims of the form "There are no X".A canonical example is the old "There are no Black Swans". This was disproved by the discovery of a species of swan that is (mostly) black. It lives in Australia, so at one time it was Unknown to Science. You can rephrase this in the positive form, "There are Black Swans", and such existence statements can be "proved" by simply presenting examples. But this is generally classified as data collection, which is understood to always be incomplete. And such negative claims are generally not taken seriously by scientists unless you can give good reasons why X can't exist, based on previously accepted theories. Even then, a single (non-fraudulent) example can suffice to shoot down your reasoned argument against X existing.

      In any case, "proof" is something done by mathematicians, not scientists. If you reject science that doesn't present proof, you reject all science, since proof isn't what science does.

      If all you have left is belief, then you are susceptible to being defrauded by anyone who comes along with a new belief. But history shows that science's testing process has been pretty good at disproving most beliefs. In the process, the leftover ("not disproved") beliefs that fell out of the process have led to all the technical advances of the previous several centuries, something that the earlier purveyors of belief systems ("religions") have failed to do for as long as we have recorded history.

  • by belthize (990217) on Sunday May 06, 2012 @12:55PM (#39909251)

    In thinking about how I look at articles and commentary I realized I factor in comments almost as much as the article itself, particularly any inherently subjective article, for example one that discusses the social or economic impact of a scientific discovery.

    The article itself is likely to have a good signal to noise but suffers from bias, the comments typically have very poor signal to noise but can often correct or at least expose the original biasing. Taken together I at least feel like I have a better sense of 'truth', particularly if the subject is likely to expose my own bias.

    In other words, yeah the article makes sense initially but I'll reserve judgment till more people have posted about it on slashdot.

  • by Sebastopol (189276) on Sunday May 06, 2012 @01:18PM (#39909419) Homepage

    Although the author set out to analyze the role comments play, I found his objective disposition of woodpecker sightings' impact on environmental fundraising. Almost bordering on cynical, the author does point out the suspicious nature of theses sightings in a way that I'm sure will ruffle the feathers of many set-in-stone environmental saviors.

  • I thought science looks for "consensus" nowadays, not "truth."

  • by Lord of the Fries (132154) on Sunday May 06, 2012 @10:05PM (#39912267) Homepage

    Is it just me, or is it quite ironic that at the time of writing this, this will be only the 47th comment on a subject active for 12 hours on slashdot. Apparently metacrowdsourcing (crowdscourcing about crowdsourcing) isn't all that popular.

    For reference sake, there have been 7 more articles up (at least by my filters) which have already garnered the following amounts of comments. Notice that the less scientific seems to be where more, er, uh "crowdsourcing" happens. :)

    ...older...
    Is Google the New Microsoft? --> 366
    Study Aims To Read Dogs' Thoughts --> 113
    Apple Security Blunder Exposes Lion Login Passwords In Clear Text -- >144
    Biochemist Creates CO2-Eating Light That Runs On Algae --> 76
    Some USAF Pilots Refuse To Fly F-22 Raptor --> 191
    Ask Slashdot: What Language Should a Former Coder Dig Into? --> 229
    Unblocking The Pirate Bay the Hard Way Is Fun --> 51

  • and different media can be used to discuss the falsifying data. So, what does "type of media" have to do with "scientific truth"? (hint: nothing) I guess the author's main point is that "crowd sourcing" is being ignored by the "elite scientific media". NYTimes, not your best moment here...
  • Consensus != Truth

  • Comment boards and science don't mix. Comment boards are the new religion, spreading FUD and turning nonsense into science simply because "enough" people have drunk the Kool-aid.

  • Appeal to authority, whether the authority is a crowd or a bearded man in a white robe, is a logical fallacy.

It's a naive, domestic operating system without any breeding, but I think you'll be amused by its presumption.

Working...