The Alignment Illusion: How BadReward Corrupts Text-to-Image Generation

| Source: arXiv AI

Tags: BadReward, RLHF, Data Poisoning, Text-to-Image, Reward Models, AI Security

BadReward shows that text-to-image models can be poisoned through the RLHF stage with visually benign preference pairs that corrupt the reward model, a newly accepted IEEE TIFS paper reports.

Details

Text-to-image models increasingly rely on RLHF to align outputs with human preferences and safety rules. Poisoning attacks on supervised fine-tuning are well studied, but the RLHF stage had not been examined as an attack surface. BadReward is an indirect clean-label attack. It uses feature collision to craft preference image pairs that look harmless but align with a malicious target concept in the reward model's feature space, corrupting the reward signal. Online RLHF amplifies the corruption through its reward feedback loop, while offline preference optimization learns it directly from the poisoned data. The authors report high attack success and visual stealth across RLHF algorithms on modern architectures. This is a revised version of a 2025 paper, now accepted to IEEE TIFS.