AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds

| Source: arXiv AI

Tags: AceAction, Vision-Language-Action, Game AI, Datasets, Hierarchical Planning

AceAction is a dataset of human gameplay across multiple 3D games with hierarchical task, subgoal and skill annotations aligned to keyboard-and-mouse actions, aimed at training generalist vision-language-action agents.

Details

Collecting vision-language-action data in the physical world is expensive, so the authors turn to 3D video games. Existing game datasets, they say, have narrow game coverage, sparse language, or temporal structure that does not track objective changes. AceAction organizes demonstrations around complete tasks, linking overall instructions, ordered subgoals and local skill instructions to aligned visual frames and keyboard-and-mouse actions. A vision-language model pipeline detects subgoals from changes in objectives or interaction targets and anchors navigation to visible waypoints. Demonstrations span navigation, reconnaissance, combat, interaction and recovery. An audit of 500 trajectories showed a 95.8% pass rate. The dataset supports research on how agents compose and switch skills.