AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment
| Source: arXiv AI
Tags: aviation AI, LLM agents, safety-critical systems, benchmarks, autonomous systems, procedural execution
AeroCopilotBench evaluates LLM agents as aviation copilots on 1,200 knowledge questions and 73 emergency procedural tasks in a virtual cockpit with hard safety constraints. The best model achieves 72.6% success, and strong static knowledge consistently fails to predict safe procedural execution.
Details
Aviation is a safety-critical domain where AI copilot assistants face two distinct challenges: domain knowledge and procedural execution under emergency conditions. Existing aviation evaluations focus on static knowledge, missing the interactive execution dimension entirely. AeroCopilotBench introduces ACOE (AeroCopilot Operational Environment), a reproducible virtual-cockpit environment that converts natural-language procedures from actual Pilot's Operating Handbooks (POHs) into executable state transitions with hard safety constraints. Tier-1 uses 1,200 multiple-choice questions. Tier-2 uses 73 emergency and abnormal tasks where an agent must diagnose faults, operate cockpit systems through standardized tool interfaces, and avoid any hard safety constraint violation — success requires all goals achieved with zero violations. Results across 12 models: best Tier-2 success rate is 72.6%. Static knowledge performance does not consistently translate into procedural execution. Analysis of 451 failed episodes from 3 representative models identifies three recurring failure patterns: procedural incompleteness, failure to use state feedback, and poor long-horizon execution management. The findings motivate state-aware agent orchestration and regression testing as key engineering priorities.