We still don’t know how people are really using AI

Details

AI companies like Anthropic and OpenAI publish usage reports, but they curate what gets included. A new independent project called the AI Observatory — co-led by Anka Reuel at Stanford's STAIR Lab — aggregated seven real conversation datasets collected with user consent to provide an unfiltered look at how people actually use AI. The findings are striking. When the Observatory team applied Anthropic's own Economic Index methodology to their dataset, they found it would exclude 48% of conversations. Those filtered-out conversations showed dramatically higher rates of sensitive content: harassment and hate (27.5% vs. 5.66% in Anthropic's analysis), sexual content (16.7% vs. 2.4%), and adult/illicit topics (7.9% vs. 2.1%). OpenAI's 2025 report similarly acknowledged that only 30% of consumer ChatGPT use is work-related. Beyond the coverage gap, the Observatory found that AI use varies significantly across models and has shifted over time. Conversations in WildChat, one of the larger datasets, grew longer and more elaborate between 2023 and 2025, with increases in prompt tokens, response tokens, and turns. More small talk emerged too. The platform is intended for researchers and policymakers making consequential decisions about AI risks and benefits based on data that, until now, came entirely from the companies with a stake in the outcome. The researchers note the Observatory provides a bird's-eye view that siloed company reports cannot replicate.