Experimenting On Mechanical Turk 46
itwbennett writes "In a recent article, Dr. Markus Jakobsson, a Principal Scientist at PARC, offers some tips on effectively running human-subject research studies on Amazon's Mechanical Turk. '...[B]enefits [include] very low experiment costs, quick turn-around rates, and relatively simple approvals from human subjects boards. But you have to be careful to avoid bias and error.' says Dr. Jakobsson. For example, in many situations subjects may be biased just from knowing that they are participating in a study, or by knowing the goals of a study. To avoid this bias, you need to 'convey a different task to your subject than what you are observing — essentially deceive them — to see how they react when faced with the situation of interest. Consider a study of user reactions to phishing sites. You may, for example, say that you are studying the common reaction to online e-commerce sites, and ask them to rate how helpful various sites are, with a free-text input field where they can add other observations. You first show them three or four legitimate websites, asking them to rate and describe them; then you show them a phishing site and do the same. Will they tell you that this is a site run by fraudsters? If they do, they noticed signs of fraud without you prompting them.'" The author also gives tips on avoiding cheaters, and determining how much to pay and when.
Man... (Score:4, Interesting)
The title makes it sounds so exciting, like we're experimenting with our robotic Turks.
But the statistician within me is also fascinated with this. It always made me wonder, as the human mind can precondition itself. The study about whether or not prayer helps the sickly followed this mindset, and since the sick humans had no idea what the study was about, when they were told that people were praying for them they thought they were much sicker and actually recovered more slowly. If they hadn't told the sick humans, however, would there have been a large difference?
Math and the brain, it's amazing how they meet.
Well now the cat is out of the bag... (Score:2, Interesting)
I often see websites where I feel there is a hidden agenda other than to make me happy enjoying the content to make the website owners some money. But people get savvy over time.. and it will start to get obvious. One thing is for sure.. you can only do this at most a few times before individuals start to figure it out, or you have get a new pool every time. Unless, you are looking for possible even more real world scenarios where there are a lot of websites that are frauds...
Huge Problem with this study (Score:3, Interesting)
If I was in a study gauging the helpfulness of various websites and one I was shown a fake website, I would simply assume they ran out of, or could not have access too the real site and so were using a mockup.
If the mockup was particularly bad I might tell them, but otherwise I'd probably chuckle to myself and then just rate it as usual.
There is a huge margin of error with this type of thinking.
Great Article, however... (Score:5, Interesting)
Great article, however you should realize it's impossible to completely avoid bias. For example, he has restricted his population to MTurk users while generalizing to the population of web users. He also "weeded out" the lazy people for the convenience of his experiment - aren't there lazy people in the real world?
Old concept, applied to the web. (Score:5, Interesting)
When I was a kid, my parents received a free episode of some potential new comedy show to evaluate for them. We watched it, commercials and all. (Yeah, the commercials were a surprise.) At the end, unsurprisingly, the questions were actually about the commercials and only the last was about the show. It didn't fool us at all, of course.
Re:Man... (Score:2, Interesting)
The study about whether or not prayer helps the sickly followed this mindset, and since the sick humans had no idea what the study was about, when they were told that people were praying for them they thought they were much sicker and actually recovered more slowly.
I've recently read about this - in anticipation of seeing Richard Dawkins speak next month. In spite of 'Christian origins' I've always considered evolution to be as rock-solid a theory as mankind has discovered... but, on reading this prominent Atheist's work, I find myself aligning more and more with the perspectives of those Dawkins dismisses as 'Creationists'. Dawkins' arguments are so unbelievably poor, it seems to me that - in effect, if not by intention - he serves as a fifth-columnist for religious cults in undermining the intellectual integrity I associate with science.
I strongly suspect that the benefit of prayer is predominantly for those who pray as opposed to those for whom prayers are directed. I definitely see benefits from 'meditation' - and expect that ritual observance is beneficial in its own right... allowing people to 'do something' when they feel helpless... it likely also has subconscious influences which are significant. Perhaps the act influences the praying to subtly alter their practical behaviour? Similarly, being prayed for by a bunch of strangers who clearly have no particular reason to care must feel depressing - the idea that "it's come to this" could easily be a negative influence - especially for those who aren't immersed in the culture.
Why is this news? (Score:2, Interesting)