Is the Era of Cheap Internet Surveys Over?

A surprising amount of modern life depends on internet surveys.

Such surveys are the bedrock of polling, market research and much of social science. They enable pollsters to forecast elections and parties to tweak their rhetoric as public opinion shifts. In the private sector, they permit companies to gauge consumers’ interest in new products. And in academia, they allow researchers to run survey experiments and gather comparative data across countries.

The key benefit of internet surveys is that they’re cheap. YouGov states on its website that it will run a “a short, nationally representative survey to 200 respondents” for $300. Running a survey the traditional way — by conducting face-to-face interviews — would cost five or ten times as much. You’d need interviewers in every part of the country, and they’d have to go out out to individual respondents’ houses.

However, the era of the cheap internet survey might be over. Why? Artificial intelligence.

As a new paper by Sean Westwood explains, AIs can now be “trivially programmed” to answer online surveys in ways that are essentially indistinguishable from humans — and not just humans in general but specific categories of respondent. You can tell an AI to pose as a ‘middle-aged white woman from California’ or as a ‘young black man from Alabama’ or anything else. And the AI will give answers that are consistent with the typical answers of those types of respondents.

Of course, online platforms have long been aware of the risk of bots or inattentive humans ‘contaminating’ their surveys. However, these risks were easily detectable. Traditional bots respond at random or just click the first option for every question. They also tend to complete surveys extremely fast, since they’re designed to fill out as many as possible and collect the small fee users receive for taking part.  

New ‘synthetic respondents’ are far more sophisticated. As Westwood demonstrates in the paper, they can bypass standard ‘attention checks’ over 99% of the time. For example, they can easily pass tests of image recognition or reading comprehension that were designed to filter out bots. And it’s unclear what other tests could be used in their place since AIs are already better at test-taking than most humans.

Why would anyone bother to program an AI to fill out online surveys? One obvious reason is money. As Westwood notes, it would only cost about $0.05 to fill out a standard survey with AI, as compared to a payout of around $1.50. So you’re looking at a profit margin of 97%.

Another potential motive is to manipulate public opinion. Foreign adversaries could program AIs to give answers more favourable to their national interests, with the aim of dissuading lawmakers in the target country from taking actions that might harm them. For example, China could create synthetic respondents that always answer ‘no’ to items like, ‘Should the US adopt a more hostile stance towards the People’s Republic?’

The only way of addressing this problem (aside from reverting to face-to-face interviews) is much more vetting of respondents: requiring proof of identification, using multi-factor authentication etc. And even those obstacles might not prove insurmountable for sufficiently motivated actors.

ChatGPT has already been commercially available for three years. And Westwood is surely not the first person to whom this idea has occurred. How much of the survey data produced since 2022 was contaminated with AI? We simply don’t know.