Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets
| Source: arXiv AI
Tags: LLM safety, jailbreak, GPT-5, red teaming, AI security, adversarial ML, evaluation methodology
Fair-ASR is a new jailbreak evaluation protocol that uses target API calls as the comparison axis — and when applied to 11 attacks, simple stochastic methods outperform complex LLM-based attacks under equal budget. ReCode, built from this analysis, achieves 85% attack success on GPT-5 with just 20 target calls.
Details
Most LLM jailbreak evaluations report attack success rate (ASR) without controlling for compute budget, making comparisons misleading: a method that requires 1000 API calls to succeed looks as good as one that succeeds in 5. Fair-ASR addresses this by standardizing evaluations under a shared target-call budget B, using direct API calls to the target model as the primary comparison axis (attacker-side compute tracked separately for efficiency analysis). Re-evaluating 11 representative jailbreak attacks under Fair-ASR reveals that rankings shift substantially across budgets: simple stochastic perturbations and hand-crafted template attacks remain highly competitive against more complex LLM-driven methods when given equal access to the target model. No tested LLM-driven method is efficient in both target calls and attacker calls simultaneously. Armed with these findings, the authors introduce ReCode — a compositional attack combining desensitization rewriting with two effective low-cost primitives identified by Fair-ASR analysis. Under a budget of 20 target calls, ReCode achieves 85% ASR on GPT-5 while requiring only 7.19 attacker calls per request. This is a significant finding: GPT-5, OpenAI's most capable model, remains highly vulnerable to carefully designed budget-efficient jailbreaks.