AI can predict how you’ll respond to a survey. But that’s not the same as understanding you
AI can predict how you’ll respond to a survey. But that’s not the same as understanding you
Publish Date: 2026-07-09 00:34:00
Source Domain: theconversation.com
What makes people change their minds, or their behaviour? Social scientists spend a lot of time thinking about this question, and experiments are one of the most powerful ways to answer it.
Experiments – testing ideas on real people – take considerable amounts of time and money. Enter large language models (LLMs): artificial intelligence (AI) systems trained to mimic certain kinds of text-based human behaviour based on vast amounts of human-produced text.
A new study led by Harvard psychology researcher Ashwini Ashokkumar, published today in Nature, suggests LLMs such as GPT-4 can predict the outcomes of many social science experiments surprisingly well.
But the results come with a warning: a system that predicts human responses is not necessarily a system that understands human behaviour, and “synthetic respondents” or “silicon samples” are not a direct substitute for real people.
A striking result
Ashokkumar and her colleagues assembled 70 real experiments already conducted in the United States involving almost 120,000 participants.
They then gave GPT-4 descriptions of hypothetical respondents alongside experimental messages and survey questions, and asked it to estimate how those people would respond under different conditions.
Next they compared GPT-4’s predictions with the real results, and found a strong correlation. The model could often distinguish between interventions that were more or less effective.
This is a striking result. It suggests LLMs may capture meaningful patterns in the social world – at least in the kinds of text-based US survey experiments examined in the study.
But it’s not evidence that AI has discovered a reliable shortcut around human research.
Useful forecasts are not the same as understanding
US scholars Lisa Messeri and Molly J. Crockett have warned that AI systems can create “illusions of understanding”: outputs that look insightful and useful, while encouraging users to overestimate what…
