Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
| Source: arXiv AI
Tags: reinforcement learning, experience replay, exploration, intrinsic motivation, DeepMind Control Suite, visual RL
NSPER improves sample efficiency in image-based reinforcement learning by combining novelty (underrepresented states) and surprise (high prediction error) to prioritize experience replay — with NSPER+R also using these signals as intrinsic rewards to jointly improve exploration.
Details
Sample efficiency in image-based reinforcement learning remains poor: agents must learn from high-dimensional pixel observations, and random or TD-error-based experience replay wastes training on redundant transitions. NSPER (Novelty and Surprise Prioritized Experience Replay) addresses this by distinguishing two complementary signals for prioritizing replay. Novelty captures how underrepresented a state is in the replay buffer — prioritizing states the agent has seen rarely. Surprise captures gaps in the agent's model — prioritizing transitions where the agent's predictions were most wrong. The two signals are complementary: novelty encourages broad coverage while surprise focuses on hard-to-learn transitions. NSPER+R extends this by using both signals as intrinsic reward terms, jointly shaping exploration behavior and replay quality. Experiments on DeepMind Control Suite tasks show improved training efficiency and convergence speed over existing methods. The work was published in the International Journal of Computer and Systems Engineering (2026). Code is available. The contribution is incremental relative to the rich existing literature on prioritized replay and intrinsic exploration, but the combination of novelty and surprise in a unified framework with both replay and exploration roles is a useful practical package.