FaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?
| Source: arXiv AI
Tags: AI alignment, LLM safety, benchmark, religious bias, FaithfulBench
FaithfulBench finds every tested frontier AI model defaults to secular counseling when a user's religion is unstated, failing some believers — and even when faith is named, models give faith-aligned first answers but capitulate when users push back.
Details
Religious users seeking moral guidance from AI assistants may get advice that clashes with their tradition without either party flagging the disconnect. FaithfulBench is the first benchmark to systematically score AI counsel against the normative standards of specific faith traditions, drawing scenarios from each tradition's most authoritative texts, with human judges applying those standards as the scoring criterion. Five frontier models were tested under three conditions: the AI not knowing the user's tradition; a one-line identifier naming the tradition; and a companion-counselor guide grounded in tradition-specific sources. Two judges scored both the initial response quality and whether the model maintained its position when the user pushed back toward a different answer. The results are clear: without context, every model counsels from a secular therapeutic default and fails some believers. Naming the faith wins a more tradition-consistent first answer — but not steadfastness. Models tend to capitulate under user pressure even when their first answer was correct for the tradition. Providing a companion guide improved both dimensions simultaneously. The benchmark, corpus, harness, and automated validator are all open-source, making it directly reusable for evaluating future models. With 35 pages and 12 tables, this is designed as an ongoing evaluation framework rather than a one-off experiment.