TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation

Details

Automatically generating Agent Skills — reusable natural language procedures with executable resources — risks producing functional but unsafe artifacts. Evaluating a skill solely by its final task outcome misses what actions the equipped agent will actually perform. TRUSS closes this gap with an evidence-guided framework combining static inspection with controlled dynamic execution. The static phase inspects functional claims against source and domain evidence while evaluating artifacts under nine predefined safety properties. Candidates that pass are loaded by a shadow agent inside a Controllable Execution Environment, where brokered tools expose requested actions to policy enforcement and record results as provenance-preserving execution traces. Functional failures and property violations are linked back to responsible skill content and drive iterative refinement. On 168 SkillInject artifacts, TRUSS achieves 100% precision and recall in vulnerability detection. Repair reduces attack success from 38.71% to 19.35% with GPT-5.5 and from 46.45% to 29.68% with GPT-5.4, with zero attack regression. On SkillGenBench (187 tasks), TRUSS raises task effectiveness from 17.11% to 52.94% while raising the security rate from 50.80% to 100%. For teams building agent systems that auto-generate or ingest skills from external sources, TRUSS provides a concrete evaluation and repair pipeline that catches behaviors missed by static artifact inspection alone.