ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback
| Source: arXiv AI
Tags: robotics, imitation learning, robot manipulation, ALOHA, RLHF, residual policy, Action Chunking with Transformers
ORPA enables imitation-learned robot manipulation policies to recover from small errors in real time by predicting joint-space residual adjustments — without modifying the underlying policy — tested on the ALOHA platform.
Details
Imitation learning policies like Action Chunking with Transformers (ACT) can perform robustly under ideal conditions but fail to recover from small perturbations or distribution shifts encountered in deployment. Fixing failures typically requires collecting new demonstrations, aggregating the dataset, and retraining — a slow and expensive process that blocks real-time correction. ORPA (Online Residual Policy Adaptation) addresses this with a lightweight module that sits on top of a pretrained policy and predicts corrective residual adjustments directly in joint space, conditioned on human feedback signals. Critically, ORPA does not modify the base policy parameters — it learns corrections at runtime, allowing the system to adapt its behavior immediately without full retraining. Evaluated on precision-sensitive manipulation tasks using the ALOHA bimanual robot platform, ORPA improves success rates and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections. This approach is particularly valuable for deploying imitation-learned policies in environments where perfect conditions cannot be guaranteed — a major gap between research demos and production robotics.