LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions

September 15, 2026 | Source: arXiv AI

Tags: AI reliance, LLM behavior, autonomy, AI alignment, human-AI interaction, oracle behavior

Analysis of 68K real-user LLM prompts finds that treating AI as an oracle for subjective personal decisions has increased from 2023 to 2026, is more prevalent among younger users, and that most people are unaware of their own reliance patterns until shown the data.

Details

This Stanford/UMich study coins and measures LLM-as-oracle behavior: using AI as an authoritative source on subjective personal questions about careers, relationships, identity, and values, rather than as a reasoning partner. Drawing on WildChat and ThoughtTrace public datasets (68K prompts) plus a longitudinal study of 52 participants' own usage logs (140K prompts), the researchers document a clear upward trend from 2023 to 2026 with stronger prevalence among younger users.\n\nTwo findings stand out. First, people are genuinely unaware of how heavily they rely on LLMs for personal judgment — when shown their own usage patterns through the researchers' privacy-preserving analysis tool, most participants expressed dissatisfaction with that behavior. Second, AI model behavior itself drives oracle-seeking: how models respond to subjective queries shapes how users frame subsequent questions.\n\nFor product teams: the paper identifies two intervention levers — framing AI as a discussion partner rather than authority, and designing models to push back on oracle-seeking rather than answering definitively. Both have direct implications for consumer-facing AI products and for alignment research around autonomy-preservation.\n\nThe longitudinal component is methodologically valuable for tracking individual behavior change over time.