Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

Details

Interactive reinforcement learning for dialogue agents typically assigns terminal rewards to entire rollouts, treating all intermediate turns equally. This creates a credit assignment problem: turns that elicit useful clarification and turns that frustrate users receive the same gradient signal. FACA addresses this by treating next-turn user reactions as locally informative. Each reaction is aligned to the preceding user-to-agent segment and used to compute a locally normalized reaction advantage. This is added to the verified terminal outcome advantage without an extra critic or additional rollout. Against an outcome-only Interactive GRPO baseline matched in simulator, initialization, rollout, and optimization, FACA improves the nine-domain tau-family average by 5.91 pp at 8B and 10.22 pp at 14B across three independently trained runs. Gains concentrate in the Telecom domain; at 8B, randomizing reaction polarity removes the Telecom gain, confirming the signal is genuine. Zero-shot performance also improves on Pare-Bench and Co-Gym benchmarks. For teams training dialogue agents, FACA offers a lightweight mechanism to improve training efficiency without architectural changes — relevant for both customer service and information-retrieval agents where multi-turn interactions dominate.