Using Twitter Data To Approximate a Telephone Survey

cremeglace writes "A team led by a computer scientist at Carnegie Mellon University has used text-analysis software to detect tweets pertaining to various issues — such as whether President Barack Obama is doing a good job — and measure the frequency of positive or negative words ranging from 'awesome' to 'sucks.' The results were surprisingly similar to traditional surveys. For example, the ratio of Twitter posts expressing either positive or negative sentiments about President Obama produced a 'job approval rating' that closely tracked the big Gallup daily poll across 2009. The analysis also produced classic economic indicators like consumer confidence." By averaging several days' worth of tweets on presidential job approval, the researchers got results that correlated 79% with daily Gallup polling. Lead researcher Noah Smith said, "The results are noisy, as are the results of polls. Opinion pollsters have learned to compensate for these distortions, while we're still trying to identify and understand the noise in our data. Given that, I'm excited that we get any signal at all from social media that correlates with the polls." Here is CMU's press release.

Demographics Anyone

[mrtwice99]

It seems that the age demographics of twitter users wouldn't be very representative of the population as a whole.

this is going to be obsolete almost immediately

[the gnat]

I'm guessing it will take no more than a month for a combination of "conservative" and "progressive" blogs to rev up their teams of dittoheads to start flooding Twitter with politically themed messages, thus totally skewing the results. Same principle as Google-bombing, I guess. As someone who already views Twitter as almost entirely content-free, I can't say I'm particularly dismayed by this possibility. . . but anything that encourages the self-absorbed political zealots of this country can't possibly be good.

Re:Demographics Anyone

[tpstigers]

Let's not forget all other demographics. Ethnicity, gender, income, employment - just to name a few. Twitter is an amazing resource, but it's hardly representative of humanity or the nation. That said - it can still yield useful data if its limitations are taken into account.

Big surprise

[Cryacin]

The noise coming from one group of twits is the same as the noise coming from another group of twits.

Film at 11.

Re:Demographics Anyone

[Yold]

not to mention economic indicators; most poor people don't have iPhones that they can tweet their every whim. Some people also don't twitter their political views. This whole thing screams selection bias.

Twitter Users =/= Average American

[Anonymous Coward]

As someone who spends a lot of their time working to uncover endogeneities in statistical analysis, I feel that analyzing Tweets will never be a viable measure of general American opinions.

Remember when The Literary Digest predicted Alf Landon would crush FDR in the 1936 presidential election based on a poll of its subscribers? Okay, you don't *remember* that, but you've probably heard of it. Same problem here.

The readers of Literary Digest were not representative of the average American in 1936.
The users of Twitter are not representative of the average American in 2010.

Twitter polling is no better than straw polling, which is usually worse than nothing.

Re:this is going to be obsolete almost immediately

[rm999]

I was thinking of a solution to the selection bias problem that I think would also help with this issue. The researchers could "profile" different users by looking at their history. New users (with little history) and frequent but consistent users (several negative messages about a candidate a day, effectively users that tweet very little useful information) can be discounted, while more dynamic users that change their opinions in interesting ways and correlate with polls can be counted more.

Pollsters often weight their results to improve accuracy, and this would be no different. It would also remove obvious attempts to influence the results.

Re:this is going to be obsolete almost immediately

[Xtifr]

anything that encourages the self-absorbed political zealots of this country can't possibly be good.

I dunno. Encouraging them to wast their time on Twitter instead of doing things that might have an actual impact on the world sounds like a pretty good idea to me! :)

Re:Demographics Anyone

[antifoidulus]

Hell even a telephone poll, provided they picked landlines out of a phone book are increasingly less representative of the population as a whole. Young people are abandoning land-line phones to go cell phone only, most of them unlisted. I wonder how the pollsters are adapting to these demographics.

Hell, I have been considering even getting rid of the phone part of the cell phone and going data only, with Skype et al, is there even any point in paying the $30 or $40 a month for voice service?

Re:Demographics Anyone

[acohen1]

Indeed. The "people who have listed phone numbers" demographic is most certainly not representative. I don't have one, neither do 3/4 or more of my friends. I think the only ones who do are homeowners, everyone else has just a cell. So there are certainly age and economic status issues here.