Steerable Cultural Preference Optimization of Reward Models

June 18, 2026 | Source: arXiv AI

Tags: RLHF, reward models, cultural alignment, pluralistic alignment, ICML 2026, PRISM, GlobalOpinionQA

Researchers propose SCPO, a reward model training algorithm that balances cultural preferences across 7 countries, lifting minority-community scores by up to 7 points on PRISM and GlobalOpinionQA datasets while requiring 280% less training data than standard fine-tuning — a concrete step toward globally inclusive LLM alignment.

Details

Most LLM alignment research has optimized for a single unified preference signal, implicitly reflecting the views of annotators from a small number of regions. SCPO (Steerable Cultural Preference Optimization), accepted at ICML 2026's Pluralistic Alignment workshop, addresses this by training reward models that can represent diverse cultural subcommunities without over-weighting the majority group. The algorithm uses a weighting mechanism that brings minority-community reward model performance up by as much as 7 points across two evaluation datasets — PRISM and GlobalOpinionQA — and across 7 countries. Crucially, it achieves this with roughly 280% greater data efficiency compared to simply fine-tuning on the full dataset for each culture. The practical implication is that a single steerable reward model could serve multiple cultural contexts without deploying separate models for each group. The code is publicly available, making this immediately usable for teams training RLHF pipelines.