JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills
| Source: arXiv AI
Tags: JailbreakSkill, red-teaming, jailbreak, AI safety, GPT-5.4, AdvBench, HarmBench
JailbreakSkill packages jailbreak attacks into modular, evolving skills — lifting attack success rates by 17.5 pp on AdvBench and achieving a 48.6-point gain against GPT-5.4, with novel attack strategies (like reframing harmful requests as document completion) emerging automatically from the feedback loop.
Details
Red-teaming AI systems has been a fragmented effort: researchers devise individual attack prompts and workflows, but these are rarely systematized or reused at scale. JailbreakSkill takes a different approach by treating attack strategies as modular, agent-ready skills that can be selected, combined, and improved automatically. The framework closes the learning loop — attack experience feeds back into skill diagnosis, refinement, and discovery, growing an ever-expanding library. The numbers are notable: macro-average attack success rate (ASR) rises by 17.5 percentage points on AdvBench and 13.4 points on HarmBench. Against GPT-5.4 specifically, the gain reaches 48.6 points on AdvBench, suggesting frontier models remain systematically attackable when attack strategies are organized and iterated. One emergent skill is particularly interesting: the framework independently discovered that reframing a harmful direct request as an unfinished document-completion task bypasses safety filters more reliably. Several evolved skills also generalize to unseen prompts and target models without additional adaptation. Code is publicly available, making this immediately useful for security teams running red-team exercises.