Oops, Not Now: PEARL, a RAG-Based Support Agent for Gameplay and What Players Want from AI Help
| Source: arXiv AI
Tags: RAG, game-based learning, HCI, AI agents, education, LLM
PEARL, a dual-component RAG agent for a programming puzzle game, was minimized or abandoned by 5 of 10 players — generic responses and proactive interruptions drove disengagement, yielding a concrete failure taxonomy for AI support system designers.
Details
AI-powered in-game support has obvious appeal for educational games, but building agents that players actually use is harder than building agents that technically work. PEARL (Parallel Education Agent for Reflection and Learning) is a RAG system deployed in Parallel, a puzzle game for learning parallel programming concepts. PEARL combines two retrieval streams: semantic search over conceptual explanations and structural board-state matching against peer-generated solutions. This gives it capabilities unavailable to a plain LLM with game state access — matching the current puzzle configuration against similar historical player states. In a qualitative evaluation with 10 participants, PEARL performed poorly against an existing visualization system. Five players minimized or abandoned the AI tool entirely during play. The identified failure modes were specific: proactive unsolicited interruptions, generic responses that didn't reflect the player's actual game state, and a trust deficit when early answers missed the mark. A subset of four participants found PEARL complementary to visualization when they initiated the interaction themselves. The paper reframes the failed deployment as a structured design probe, deriving seven open design problems for the AI-in-games community — covering proactivity, trust calibration, grounding, and when to stay silent.