Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps
| Source: Apple ML Research
Tags: Apple ML Research, PPO, GiGPPO, LLM-as-a-Judge, iOSWorld, AndroidWorld, UI navigation agents
Apple researchers trained 8B UI-navigation agents with online reinforcement learning on hundreds of live virtual iOS devices, matching Gemini-3.1-Pro on real iOS apps and cutting the cost case for frontier models in on-device automation.
Details
Apple ML Research describes an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications instead of static trajectories, which tend to make small SFT-trained models brittle. The agent is split into a Planning agent and a UI Grounding model. An LLM-as-a-Judge supplies dense step-level rewards plus a terminal task-completion reward, optimized with PPO and step-level advantage estimation. A new method, GiGPPO, stabilizes the critic by sharing reward signals across visually similar states in concurrent trajectories. The team built infrastructure orchestrating hundreds of concurrent virtual iOS devices. On UINavBench, iOSWorld and AndroidWorld, the online RL agents beat SFT and terminal-only RL baselines and converged faster. The 8B agents showed nearly a sevenfold gain over SFT in out-of-distribution transfer to Android, outperformed Gemini-3.5-Flash and tied Gemini-3.1-Pro on real iOS apps.