从演示到奖励：VLM奖励模型的测试时提示优化
From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
AI总结：
提出Demo2Reward方法，利用少量专家演示在测试时优化VLM奖励模型的提示指令，减少假阳性并保持真阳性，无需额外训练即可提升下游策略学习。
摘要：
强化学习依赖于准确的奖励函数，但在现实应用（如机器人技术）中，这些函数通常是手工设计的，甚至不可用。最近的研究探索了预训练视觉-语言模型（VLM）作为奖励模型的零样本推理能力。然而，如果没有仔细的提示工程，这些方法往往会产生次优的奖励，其中假阳性预测会严重降低下游策略学习。在机器人技术中，通常收集包含专家演示的有限数据集来引导策略学习。这种场景提供了在策略训练之前优化奖励模型的机会。我们提出Demo2Reward，一种测试时自适应技术，基于少量演示（3-10条轨迹）优化奖励模型的语言指令，以减少假阳性同时保持真阳性。关键是，这在策略学习期间不需要额外的模型训练或计算资源。我们表明，Demo2Reward在一系列模拟机器人任务和策略骨干上始终优于现有的零样本和少样本VLM奖励模型。最后，我们证明Demo2Reward有效迁移到真实世界的机器人学习场景，无需手动设计奖励函数即可实现策略学习。
英文摘要：
Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
