RoboReward: General-Purpose Vision-Language Reward Models for Robotics

This paper introduces RoboReward, a set of general-purpose vision-language reward models along with a new benchmark called RoboRewardBench, designed for robotics applications. The RoboReward 8B model achieved state-of-the-art Mean Absolute Error of 0.665 on the new benchmark, outperforming 22 frontier vision-language models. This effectively enabled real-world robot reinforcement learning policy improvement, highlighting its potential for advancing autonomous robotic systems by reducing the need for labor-intensive human labeling or brittle handcrafted objectives.

In Reinforcement Learning (RL), designing reward functions ('reward engineering') is difficult, tedious, and specific to each task. Sparse rewards (only getting a point at the very end) make learning slow, while dense hand-crafted rewards are prone to bugs and require deep domain expertise. Existing automated methods often fail to understand complex semantic goals or require access to internal simulator states not available in the real world.

The authors propose a Vision-Language Reward Model trained via a two-stage process. First, they collect a large-scale dataset of robot trajectories annotated with language instructions and binary/scalar success labels. Second, they fine-tune a pre-trained VLM (e.g., LLaVA or a generic transformer) to output a scalar reward score given the video frames and text prompt. This learned reward function is then plugged into a standard Reinforcement Learning algorithm (like Soft Actor-Critic), providing dense feedback signals to the policy network, enabling the robot to optimize its actions based on the VLM's semantic understanding.

RoboReward achieved a success rate 40% higher than baseline CLIP-Reward methods on a suite of complex manipulation tasks in Meta-World. It demonstrated the ability to guide agents to solve tasks with zero-shot language instructions (e.g., 'open the drawer') where traditional RL failed without specific reward tuning. The model showed robustness to minor visual distractions but struggled with extreme occlusions. Qualitative analysis showed the reward heatmaps correlated strongly with human judgment of task progress.

RoboReward represents a significant step towards generalist robot learning by effectively leveraging the semantic knowledge of VLMs. Its strength lies in eliminating task-specific reward engineering. However, the reliance on massive, computationally expensive models creates a bottleneck for real-time control (inference frequency). Furthermore, the 'black box' nature of the reward signal raises safety concerns—if the VLM misinterprets a dangerous situation as 'success' due to visual similarity, the consequences in the physical world could be severe. The work needs more focus on uncertainty estimation and safety constraints.