Environment-free Synthetic Data Generation for API-Calling Agents

Details

Training LLM agents to call real-world APIs has historically required fully implemented sandbox environments with executable APIs and realistic pre-populated databases — a significant engineering bottleneck limiting who can create training data at scale. Apple ML Research's new method sidesteps this entirely. Given only API specifications, their three-stage pipeline uses LLMs throughout: first, a generator creates diverse tasks solvable with the provided APIs; then a teacher agent iteratively solves each task while an LLM simulator generates coherent synthetic API responses conditioned on task context and simulation history; finally, an LLM judge filters trajectories for quality. No real environment or backend database is required at any point. The method was evaluated on AppWorld and OfficeBench — challenging benchmarks spanning information-retrieval and state-changing tasks — with fine-tuning on synthetic trajectories producing significant performance gains. The paper establishes LLM-based API simulation as a practical, scalable path for training agents across diverse API ecosystems. For teams building enterprise AI agents that need to interact with internal APIs, this could substantially reduce the cost of creating fine-tuning datasets. The key risk is accuracy drift when simulated responses diverge from real API behavior — the paper does not quantify this gap, so real-environment validation remains essential before production deployment.