Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency

Details

LLM-based automated program repair (APR) is maturing, but rigorous empirical comparisons across bug complexity, fault localization precision, and cost are scarce. This ESEM 2026 paper fills that gap with a multi-dimensional empirical framework covering two APR techniques (ChatRepair and CodeCorrector) and three LLMs (DeepSeek, GPT, Llama). The most actionable finding: over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. For practitioners evaluating whether to invest in expensive models, this means a tiered strategy — try cheap models first — is justified for the majority of real-world defects. GPT-5 outperforms DeepSeek-V4-pro on complex bugs by 7 and DeepSeek-V3.2 by 39 bugs, but at significantly higher cost. DeepSeek-V3.2 shows the best overall cost-efficiency, suggesting it hits a practical sweet spot. Notably, imprecise fault localization — not knowing exactly which lines to target — amplifies performance gaps between APR techniques more than model capability differences do. Higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, challenging the assumption that throwing more powerful models at bug repair always pays off. The study evaluates across diverse bug complexity levels and localization strategies, providing cleaner evidence than prior single-dimensional comparisons.