Experimenting On Mechanical Turk 46
itwbennett writes "In a recent article, Dr. Markus Jakobsson, a Principal Scientist at PARC, offers some tips on effectively running human-subject research studies on Amazon's Mechanical Turk. '...[B]enefits [include] very low experiment costs, quick turn-around rates, and relatively simple approvals from human subjects boards. But you have to be careful to avoid bias and error.' says Dr. Jakobsson. For example, in many situations subjects may be biased just from knowing that they are participating in a study, or by knowing the goals of a study. To avoid this bias, you need to 'convey a different task to your subject than what you are observing — essentially deceive them — to see how they react when faced with the situation of interest. Consider a study of user reactions to phishing sites. You may, for example, say that you are studying the common reaction to online e-commerce sites, and ask them to rate how helpful various sites are, with a free-text input field where they can add other observations. You first show them three or four legitimate websites, asking them to rate and describe them; then you show them a phishing site and do the same. Will they tell you that this is a site run by fraudsters? If they do, they noticed signs of fraud without you prompting them.'" The author also gives tips on avoiding cheaters, and determining how much to pay and when.
I'm still waiting... (Score:3, Funny)
Man... (Score:4, Interesting)
The title makes it sounds so exciting, like we're experimenting with our robotic Turks.
But the statistician within me is also fascinated with this. It always made me wonder, as the human mind can precondition itself. The study about whether or not prayer helps the sickly followed this mindset, and since the sick humans had no idea what the study was about, when they were told that people were praying for them they thought they were much sicker and actually recovered more slowly. If they hadn't told the sick humans, however, would there have been a large difference?
Math and the brain, it's amazing how they meet.
Re:Man... (Score:5, Funny)
They better be married then!
Re: (Score:1, Offtopic)
Wow.
I mean... just wow.
If there is anyone on Slashdot who is deserving, in the extreme, of their own criticism, it's you.
I mean... just wow.
Seeing as you clearly don't have a firm grasp of the English language, it is probably best in the future that you don't start off any arguments critiquing valid grammar choices. Being a non-native speaker is not a reasonable defense for you right now.
Re: (Score:2)
Which is why I'm a skinny, semi-fit woman that has never sued anyone, is atheist, and pretty high up there in IQ-skill (if you believe that arbitrary system). Man I love when people throw broad sweeping generalizations at someone and miss the mark on each one.
What's that you say? I didn't address the "math" issue?
What's that? Every language has subtle nuanced differences, such as the various forms of Chinese, French, and Spanish to name a few? Geographical location alters the way in which we speak? Some his
Re: (Score:2)
Re: (Score:3, Funny)
When I was a stupid kid I worked at McDonalds : /
But i did walk two miles there and back everyday after school! Bought a huge trampoline with the first paycheck :) Then later a rusted pile of metal i mistook for a car... oh the memories.
Re: (Score:2)
God Answers Prayers Of Paralyzed Little Boy [theonion.com]
apropos of nothing having to do with mech turks (Score:2)
Some funny shit right there. I also like the story about the third amendment rights group.
Re: (Score:2, Interesting)
The study about whether or not prayer helps the sickly followed this mindset, and since the sick humans had no idea what the study was about, when they were told that people were praying for them they thought they were much sicker and actually recovered more slowly.
I've recently read about this - in anticipation of seeing Richard Dawkins speak next month. In spite of 'Christian origins' I've always considered evolution to be as rock-solid a theory as mankind has discovered... but, on reading this prominent Atheist's work, I find myself aligning more and more with the perspectives of those Dawkins dismisses as 'Creationists'. Dawkins' arguments are so unbelievably poor, it seems to me that - in effect, if not by intention - he serves as a fifth-columnist for religiou
Re: (Score:2)
Meditation is a lot about clearing your mind and simply relaxing, which does the body a lot of good. Praying makes us feel good about ourselves because we feel we've done something. Maybe then we should assume that prayer and the like does not have divine power, but that the sense of relaxation or helping someone else makes us feel better.
Now there's a study worth doing. Another study worth doing would be to see if loved ones praying for someone without that someone knowing would indeed speed up their recov
Re: (Score:2)
LOL, very Dawkinsian. The problem with the divine is that, by definition, it defies scientific investigation. Sure, you might be able to set up these experiments - but you won't be able to overcome my objection that correlation is not causation.
This relates to other ideas I've been pondering of late (of a more earthly variety...) relating to economic policy. Essentially, central bankers are faced with something of a 'divine' problem in the task they're set. There are 'levers' that can be pulled, and the
Re:Man... (Score:4, Funny)
Math and the brain, it's amazing how they meet.
(Brain is walking past dark alley)
Math: Pssst!! Hey, you!
Brain: Huh? Wh-- me?
Math: Yeah - you! You wanna try some good shit? I got some seriously advanced Number Theory, here. You totally have to try this shit!
Brain: Nah, thanks anyway, man, but I'm not into that hard shit.
Math: You sure?! This shit's fucking amazing, man. Real pure, man, grade A. Some of this shit will blow your fucking mind!
Brain: Nah it's cool, thank--
Math: That's cool, that's cool, man. It's not for everybody...how 'bout some primo Euclidean Geometry? You ain't never calculated the sum of the squares of the sides of a right triangle until you've demonstrated the Pythagorean Theorem. This shit's a trip!
Brain: Hey, yeah, I'd be down for some of that...
Math: Excellent! Now don't worry about nothin' - the first book is on me... friend. (puts his arm around Brain)
Brain: Hey, man, you're all right, y'know...
Well now the cat is out of the bag... (Score:2, Interesting)
I often see websites where I feel there is a hidden agenda other than to make me happy enjoying the content to make the website owners some money. But people get savvy over time.. and it will start to get obvious. One thing is for sure.. you can only do this at most a few times before individuals start to figure it out, or you have get a new pool every time. Unless, you are looking for possible even more real world scenarios where there are a lot of websites that are frauds...
Re:Well now the cat is out of the bag... (Score:4, Insightful)
No, with most people you can do it over, and over, and over.
It's why Fox News makes money.
Huge Problem with this study (Score:3, Interesting)
If I was in a study gauging the helpfulness of various websites and one I was shown a fake website, I would simply assume they ran out of, or could not have access too the real site and so were using a mockup.
If the mockup was particularly bad I might tell them, but otherwise I'd probably chuckle to myself and then just rate it as usual.
There is a huge margin of error with this type of thinking.
you may want to scrub. (Score:2, Funny)
i make my mechanical Turk have sex with my mechanical JD.
Great Article, however... (Score:5, Interesting)
Great article, however you should realize it's impossible to completely avoid bias. For example, he has restricted his population to MTurk users while generalizing to the population of web users. He also "weeded out" the lazy people for the convenience of his experiment - aren't there lazy people in the real world?
Re:Great Article, however... (Score:5, Insightful)
Great article, however you should realize it's impossible to completely avoid bias.
It's also impossible to completely avoid banal truisms masquerading as insight, but that doesn't stop us from trying.
Re: (Score:2)
Re: (Score:1)
Awww...don't be a hater. The whole article is on avoiding bias - don't you see a little irony here?
Better idea: take a research methods class (Score:5, Insightful)
The subject basically says it all. If you're conducting a study, either for academia or industry, do yourself a favor and take a good research methods class. I can't count then number of promising studies I've rejected for publication because their methods were poor. While Dr. Jacobson makes some good points, most of them are pretty obvious to anyone who has taken a good class on creating experiments with humans -- for example, deception is a cornerstone of many human studies. Also, for you budding young scientists, make sure you get IRB approval before conducting your study. I reviewed a paper where the authors were clearly from a University and I had questions about the ethics of their methods so I asked for the IRB data for the study...whoops, they never got it. The paper was withdrawn by the authors shortly after.
MPU Re:Better idea: take a research methods class (Score:1)
Thank you for saying this. IRBs exist for a reason and it's concerning to see an article like this essentially saying "Hey kids! Go do some human experimentation!"
Re: (Score:1)
I did not find the article trivial like that. There's a lot of useful thinking the author has put into Turk specifically---and how it intersects with research methods, IRB, etc. For instance his own experiments to help think about what to pay people. (Not a lot of detail given on how he did this, but helpful information nonetheless.) Also helpful technical
Re: (Score:3, Informative)
I've done 'turking' a few times - mainly transcription work and some paragraph writing. For an individual, whether a task is worth doing really depends on how quickly they can do it. The going rate would be around $20/hour.
Simple quick repetitive tasks pay the least, while long creative tasks pay the most.
Quick tasks might just pay 10 cents and require the 'turker' to look at an image of a road from a car and click yes/no buttons to say whether there are road markings or traffic lights. Some people are good
Re: (Score:2)
Old concept, applied to the web. (Score:5, Interesting)
When I was a kid, my parents received a free episode of some potential new comedy show to evaluate for them. We watched it, commercials and all. (Yeah, the commercials were a surprise.) At the end, unsurprisingly, the questions were actually about the commercials and only the last was about the show. It didn't fool us at all, of course.
Why is this news? (Score:2, Interesting)
What the hell is mechanical turk? (Score:2)
Amazon Mechanical Turk, or MTurk for short, is a cloud computing platform that permits outsourcing of tasks to other users, using a built-in payment scheme to compensate users. People (often referred to as "Turkers") perform MTurk tasks, which are called Human Intelligence Tasks (HITs), and are paid just a few cents for completing them.
Although I'm still not sure what is mechanical or turkish about it. The Amazon part apparently refers to the fact that payment is made in way of credits to Amazon.com.
Re: (Score:2, Informative)
...
Although I'm still not sure what is mechanical or turkish about it. The Amazon part apparently refers to the fact that payment is made in way of credits to Amazon.com.
Amazon got the name from this: http://en.wikipedia.org/wiki/The_Turk [wikipedia.org]
Biased in the other direction (Score:1)
There've been enough comments about why you need to trick your participants, so I won't go into it, but the specific content of this study made me think.
I usually go to websites thinking that they are probably going to scam me, and don't let my guard down until they've demonstrated that they aren't. But in this situation, where I'm being directed to it as part of a survey on e-commerce sites, I'd probably show a lot less scepticism at the start, and would be less likely to flag it as if I had just stumbled
What's a Mechanical Turk? (Score:3, Funny)
Is it something like an Electric Monk?
+1 to the /. crew... (Score:1)
...for spotting the chess reference in the name.
No IRB? (Score:2)
Contrary to the article (at CMU at least) we _are_ required to run our Mturk studies past the IRB.
They all get rubber stamped 99% of the time, but it's still an important formality.