Evaluating Agentic Code Repair Capabilities in Distributed Systems

Details

LLM coding agents have rapidly improved on single-process debugging: frontier models now cluster near the ceiling of SWE-bench Verified. DDBench asks whether those gains transfer to distributed systems, where bugs span processes, nodes, and protocol interactions and root causes are rarely visible from source code alone.\n\nThe benchmark covers 60 historical bugs from 13 open-source distributed systems, divided into three difficulty tiers. Each bug is evaluated under two conditions: symptom-only (agent sees only the bug description and repo) and context-augmented (agent also receives logs, runtime traces, and targeted code-investigation notes). This design isolates the effect of debugging context from raw model capability.\n\nKey findings: on the hardest tier, model pass rates span 61 percentage points — a much wider spread than on SWE-bench, indicating distributed debugging actually differentiates models. Adding bounded debugging context lifts aggregate pass rate by 18.1 points. The lift is asymmetric: weaker models gain pass rate; stronger models gain efficiency (solving the same bugs with fewer attempts). Context can also mislead — even faithful debugging information sometimes sends models in the wrong direction, suggesting context curation is as important as context provision.\n\nFor practitioners building AI coding agents or evaluating deployment in distributed systems contexts, DDBench offers a more realistic challenge than existing benchmarks and concrete design guidance: structured debugging context is worth roughly 18 points, but its quality matters.