RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation

| Source: arXiv AI

Tags: RoboScope Kit, LIBERO, RoboTwin, StarVLA, CUDA MPS, robot policy evaluation

RoboScope Kit wraps robot policies and benchmarks behind shared interfaces and smarter scheduling, cutting 500-episode RoboTwin evaluation from about six hours to 14.4 minutes on four GPUs with identical outcomes.

Details

Robot policy benchmarks each come with their own evaluator and conventions, and they are slow: 500 RoboTwin Easy episodes on one GPU took 359.7 minutes, while profiling the StarVLA evaluator showed only 14.3% average GPU utilization. RoboScope Kit wraps each model once as a Policy class and each benchmark once in an adapter around its own evaluator, so rules and scoring stay unchanged. It runs simulator workers and Policy replicas as separate pools, sizes them from measured probes and shards work across GPUs. Adapters cover six benchmark families. On LIBERO-Long with one RTX 4090, it ran 4.45x faster than the official StarVLA evaluation, 2.78x faster than RLinf and 1.52x faster than vla-eval, rising to 6.16x with CUDA MPS. On RoboTwin Easy, four GPUs finished 500 episodes in 14.4 minutes, 25.04x faster than the single-GPU baseline.