Benchmarking Automated Security Patch Backporting: How Far Are We?
| Source: arXiv AI
Tags: security patching, LLM agents, vulnerability management, software engineering, benchmarking
Porting Benchmark exposes a sharp gap in automated security patch backporting: the best tool drops from 85.2% success on simple patches to 24.0% on complex ones — and executable validation reveals failures that reference-match scoring misses entirely.
Details
Automated security patch backporting — applying fixes from newer code to older maintained branches — is a critical N-day vulnerability management task. Existing tools claim 80%+ success rates, but those benchmarks use narrow, homogeneous datasets. Porting Benchmark introduces 1,234 cases spanning cross-version, cross-branch, and cross-repository scenarios under a common evaluation framework.\n\nFive tools are evaluated: PortGPT, TSBPort, FixMorph, Mystique, and others spanning program analysis, LLM prompting, and LLM agents. Under aligned evaluation, PortGPT and TSBPort remain comparatively strong on a Replication Dataset, while FixMorph and Mystique degrade substantially. On structurally complex patches (Type-IV), the best commit-level success rate falls from 85.2% to 24.0% — a dramatic performance cliff.\n\nA particularly important methodological finding: exact-match reference scoring under-credits harder adaptations and misses residual integration failures. A 45-case dynamically validated subset with verified test cases and constructed proof-of-concept exploits shows that static reference agreement doesn't capture real-world remediation success. Executable-feedback refinement provides limited but measurable recovery on the hardest cases.\n\nFour root-cause categories are identified: missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure. Accepted at ASE 2026.