A Teenager on TikTok Disrupted Thousands of Scientific Studies With a Single Video (theverge.com) 49
Thousands of scientific studies had to toss out weeks of data because of a 56-second TikTok video by a teenager. From a report: The July 23rd video is short and simple. It opens with recent Florida high school graduate and self-described "teen author" Sarah Frank sitting in her bedroom and smiling at the camera. "Welcome to side hustles I recommend trying -- part one," she says in the video, pointing users to the website Prolific.co. "Basically, it's a bunch of surveys for different amounts of money and different amounts of time." That video got 4.1 million views in the month after it was posted and sent tens of thousands of new users flooding to the Prolific platform. Prolific, a tool for scientists conducting behavioral research, had no screening tools in place to make sure that it delivered representative population samples to each study. Suddenly, scientists used to getting a wide mix of subjects for their Prolific studies saw their surveys flooded with responses from young women around Frank's age.
Though not particularly well known, Prolific is part of a small collection of online tools that have transformed the way corporations and scientists study the way people think and act. The first and largest of these research platforms is Amazon-owned Mechanical Turk, which was released in 2005 as a general-purpose platform for crowdsourcing work on repetitive tasks. Soon after it was released, behavioral scientists realized its potential value for their research, and it quickly revolutionized several research fields. [...] The Behavioral Lab at Stanford mainly uses the newer, smaller Prolific platform for online studies these days, said Nicholas Hall, director of the Behavioral Lab at the Stanford School of Business. While many Mechanical Turk customers are big businesses conducting corporate research, Prolific gears its product to scientists.
The smaller platform offers more transparency, promises to treat survey participants more ethically, and promises higher-quality research subjects than alternative platforms like Mechanical Turk. Scientists doing this sort of research in the United States generally want a pool of subjects who speak English as a first language, are not too practiced at taking psychological surveys, and together make up a reasonably representative demographic sample of the American population. Prolific, most agreed, did a good job providing high-quality subjects. The sudden change in the platform's demographics threatened to upend that reputation. In the days and weeks after Frank posted her video, researchers scrambled to figure out what was happening to their studies. A member of the Stanford Behavioral Laboratory posted on a Prolific forum, "we have noticed a huge leap in the number of participants on the platform in the US Pool, from 40k to 80k. Which is great, however, now a lot of our studies have a gender skew where maybe 85% of participants are women. Plus the age has been averaging around 21."
Though not particularly well known, Prolific is part of a small collection of online tools that have transformed the way corporations and scientists study the way people think and act. The first and largest of these research platforms is Amazon-owned Mechanical Turk, which was released in 2005 as a general-purpose platform for crowdsourcing work on repetitive tasks. Soon after it was released, behavioral scientists realized its potential value for their research, and it quickly revolutionized several research fields. [...] The Behavioral Lab at Stanford mainly uses the newer, smaller Prolific platform for online studies these days, said Nicholas Hall, director of the Behavioral Lab at the Stanford School of Business. While many Mechanical Turk customers are big businesses conducting corporate research, Prolific gears its product to scientists.
The smaller platform offers more transparency, promises to treat survey participants more ethically, and promises higher-quality research subjects than alternative platforms like Mechanical Turk. Scientists doing this sort of research in the United States generally want a pool of subjects who speak English as a first language, are not too practiced at taking psychological surveys, and together make up a reasonably representative demographic sample of the American population. Prolific, most agreed, did a good job providing high-quality subjects. The sudden change in the platform's demographics threatened to upend that reputation. In the days and weeks after Frank posted her video, researchers scrambled to figure out what was happening to their studies. A member of the Stanford Behavioral Laboratory posted on a Prolific forum, "we have noticed a huge leap in the number of participants on the platform in the US Pool, from 40k to 80k. Which is great, however, now a lot of our studies have a gender skew where maybe 85% of participants are women. Plus the age has been averaging around 21."
Social science...isn't (Score:2, Insightful)
Re: (Score:3)
Is social science a real science? [quora.com]
Re: (Score:2)
> unless you hold a gun to their heads
People will say anything you want when threatened with their lives and/or tortured. That is not an improvement. Though I think the whole concept of these crowd sourced surveys skews toward certain types of people. Like broke young people that want to earn a Starbucks card - nothing like being on TikTok in Starbucks while getting paid, because its all about appearances even if you live in a box.
Re:Social science...isn't (Score:5, Insightful)
You take this into account, even in surveys not related to social science. It's a part of the math. Part of the annoying math that made me regret taking statistics from the mathematics department instead of from a science department :-) Everyone knows that some percentage of results are inaccurate.
Re: (Score:2)
Re:Social science...isn't (Score:4, Interesting)
You misunderstand survey data and how it (should) be analyzed. If I ask you how you're planning to get to work tomorrow and you say "by private aircraft", I'll accept that as your response, but not necessarily 100% truth. It's what you said and statistically, we know how accurate people are in predicting their own behaviors, how frequently plans inadvertently change, and also how frequently certain demographics outright BS surveys.
If you follow the methodology, it's science. It just so happens that there are so many confounding variables when studying humans that the analysis has a wider margin of error than chemistry.
Re:Social science...isn't (Score:5, Insightful)
Social Science is a real science, however getting good data has a high degree of variance is difficult.
However much of that mess can be cleared up with Math and Statistics. Much like how here TIkTok video causes a bunch of problems, because the Math and Stats shows that the data messing up the averages was faulty.
Re: (Score:1)
Social Science is not science, it's political activism pretending to be a science through the misuse of statistical tools and the fabrication of fraudulent data and results. You can get anything published in the social "sciences", even absolute gibberish or chapters of Mein Kampf, as long as it superficially sounds like it's pushing the right agenda.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Oh the Humanities!
Re:Social science...isn't (Score:5, Informative)
Re: (Score:1)
What has no place in scientific discourse are entire fields where you can get anything from chapters of mein kampf to literal word salad gibberish published as long as it superficially sounds like it's pushing the right political agenda. Social "sciences" are not science. They are not based on any objective measure of empirical facts, they are political activism masquerading as a science through the fraudulent misuse of statistical tools and fabrication of data.
Re: (Score:3, Insightful)
Just because measuring is messy doesn't mean it's not science; it just means it's difficult science. Similarly, just because we cannot replicate the big bang doesn't mean studying it is "not science". It just means theories are harder to validate.
Re: (Score:1)
It's not like nobody designs studies to discover revealed preference.
Some social science is better science than string theory, for instance. Some is complete crap.
Beware of anybody speaking in total absolutes.
Re: (Score:2)
Social science isn't science. If you're surveying people, you don't actually have data. Unless you hold a gun to their heads, the respondents are self-selected. Some will lie, most will distort their answers. It's not science.
Wow, so much wrong with that. There's nothing wrong with self-selecting respondents. There's nothing wrong with lying respondents. The important part and the part which very much makes it a "science" is identifying how this biases your results and correcting for them.
It's not science. The TikTok video just made this obvious.
Actually false. The TikTok video just made it obvious that it is science since the video skewed the results and as such the scientific community stops to correct / abandon the data. The fact that they were able to tell their results skewed sudd
Next up, (Score:3)
Sweetwater scientists using /. poll results as a source of data.
Not only that. (Score:1)
A member of the Stanford Behavioral Laboratory posted on a Prolific forum, "we have noticed a huge leap in the number of participants on the platform in the US Pool, from 40k to 80k. Which is great, however, now a lot of our studies have a gender skew where maybe 85% of participants are women. Plus the age has been averaging around 21."
Bet the race of respondents is also now heavily skewed toward whites. Sarah Frank is white. TikTok's recommendation algorithm was accused of matching viewer and author's race last year, at a high enough priority to override other preferences and demographic matches. (The rhetoric was 'TikTok is racist' when mostly it's just segregationist.)
oh noes (Score:2)
a tool for scientists[sic] conducting behavioral research, had no screening tools in place to make sure that it delivered representative population samples to each study.
They mussed erp my surience!
Bias inherent anyway (Score:5, Interesting)
Surely they are skewewd by recruiting people who are likely to take a survey. Those people are already weird.
Poor Prolific... (Score:4, Funny)
Now Slashdot's gone and done the same thing to them again... flooding them with yet another particular demographic; the Slashdot readership
Re: (Score:2)
Re:Poor Prolific... (Score:5, Insightful)
This is how you get results like Boaty McBoatface.
Re: (Score:2)
Survey designs are tough (Score:4, Insightful)
You have to select a properly randomized sample. And account for non responses in the resulting data.
'Open' surveys are a joke. Too many times, surveys are created and targeted at politically sympathetic audiences. Only to have the news of the survey leaked to the opposition and then they all pile on to swing the results to suit themselves. There are actually companies that offer services to target public opinion surveys at selected audiences. IMO, the results provided by such firms are complete garbage.
From the sounds of it, Prolific.co was offering such 'open' surveys. With no mechanism in place to ensure proper random sampling and to handle interest groups (or just random teenagers) from biasing the samples.
Re: (Score:1)
Re: (Score:1)
Best for what?
Best in almost all respects - easy for researchers to target slices of audience demographics that they want. For respondents, they pay a minimum of £5.00 per hour. Respondents are also pre-screened - once you get to take a survey there is zero chance of being "screened out" halfway through it. They also have a very strict verification process - video selfie with ID verification, ISP whitelists etc. - so minimizing the risk that your respondents are sat in a call centre in Gurugram or Manila masqueradi
Re: (Score:1)
Re: (Score:1)
Ask too many questions and people might suspect that the results can be de-anonymized. And refuse to respond if it involves a politically or emotionally sensitive subject
Prolific guard against this too - they are the sole custodians of respondents' data. Researchers only see an anonymised id. As they operate in the UK, privacy protection is orders of magnitude better than what is legally required from a US corporation as they need to be GDPR compliant.
Science yay! (Score:2)
We did find something new from science... young women are airheads who waste all their time on TikTok.
Not the fault of that young lady (Score:2)
Lazy and clearly stupid "scientists" did it all to themselves.
what sort of shitty studies are they? (Score:2)
If they're already using unverified, non demonstrative samples, self-chosen 'participators'?
Your data wasn't corrupted; your shitty data sources were REVEALED.
Joke's on them (Score:2)
Microsoft already learned this the hard way (Score:2)
so, to be clear... (Score:2)
a fake science, that pretends to be studying people and how they behave, sets up a site on the internet where random people can inject some input, and then the "researchers" get surprised/flummoxed/bamboozled/[pick your related word] by the behavior of a bunch of young women...
Let that sink in for a moment...
Are YOU surprised this turned out as it did?
Then consider that these people are supposedly your superiors, with science degrees and jobs in science - the sort of "experts" that the elites of society ins