Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups
| Source: arXiv AI
Tags: AI alignment, LLM evaluation, GPT-5.2, Mistral, news framing, demographic bias, sentiment analysis
A large-scale study using 3,011 representative UK adults and 7 LLMs finds GPT-5.2 achieves 0.789 correlation with human emotional responses to political news headlines while Mistral Large 2512 scores only 0.4—leading models align broadly across demographic groups, but statistically significant inter-group differences persist even at high aggregate scores.
Details
This study addresses a specific alignment question: do LLMs match human emotional perception of news framing? The methodology is rigorous—3,011 UK adults from a demographically representative YouGov sample, plus seven LLMs, evaluated whether news headlines evoked sympathy for a specified side in political and geopolitical conflicts. GPT-5.2 achieves the highest correlation with human judgment at 0.789. Mistral Large 2512 performs worst at 0.4, illustrating substantial variation across current models on emotional alignment tasks. The top-performing models show broad alignment across demographic subgroups including age, gender, education, prior geopolitical knowledge, and participants' political predispositions. However, the paper highlights a key nuance: statistically significant differences between demographic groups exist even when aggregate performance is high. This points to the problem of differential alignment—AI systems may calibrate well to aggregate human responses while systematically misaligning with specific demographic communities. The study represents one of the most demographically robust AI alignment evaluations conducted to date, moving beyond convenience samples. For developers evaluating deployed LLMs, the cross-model comparison and demographic breakdown provide actionable calibration data.