Experimenting On Mechanical Turk 46
itwbennett writes "In a recent article, Dr. Markus Jakobsson, a Principal Scientist at PARC, offers some tips on effectively running human-subject research studies on Amazon's Mechanical Turk. '...[B]enefits [include] very low experiment costs, quick turn-around rates, and relatively simple approvals from human subjects boards. But you have to be careful to avoid bias and error.' says Dr. Jakobsson. For example, in many situations subjects may be biased just from knowing that they are participating in a study, or by knowing the goals of a study. To avoid this bias, you need to 'convey a different task to your subject than what you are observing — essentially deceive them — to see how they react when faced with the situation of interest. Consider a study of user reactions to phishing sites. You may, for example, say that you are studying the common reaction to online e-commerce sites, and ask them to rate how helpful various sites are, with a free-text input field where they can add other observations. You first show them three or four legitimate websites, asking them to rate and describe them; then you show them a phishing site and do the same. Will they tell you that this is a site run by fraudsters? If they do, they noticed signs of fraud without you prompting them.'" The author also gives tips on avoiding cheaters, and determining how much to pay and when.
Better idea: take a research methods class (Score:5, Insightful)
The subject basically says it all. If you're conducting a study, either for academia or industry, do yourself a favor and take a good research methods class. I can't count then number of promising studies I've rejected for publication because their methods were poor. While Dr. Jacobson makes some good points, most of them are pretty obvious to anyone who has taken a good class on creating experiments with humans -- for example, deception is a cornerstone of many human studies. Also, for you budding young scientists, make sure you get IRB approval before conducting your study. I reviewed a paper where the authors were clearly from a University and I had questions about the ethics of their methods so I asked for the IRB data for the study...whoops, they never got it. The paper was withdrawn by the authors shortly after.
Re:Great Article, however... (Score:5, Insightful)
Great article, however you should realize it's impossible to completely avoid bias.
It's also impossible to completely avoid banal truisms masquerading as insight, but that doesn't stop us from trying.
Re:Well now the cat is out of the bag... (Score:4, Insightful)
No, with most people you can do it over, and over, and over.
It's why Fox News makes money.