VARP：基于视觉语言模型反馈与智能体正则化偏好的强化学习
VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
AI总结：
针对机器人连续控制中奖励设计偏差与VLM偏好标注不准的问题，提出VARP方法，通过轨迹叠加提升VLM偏好准确率，结合智能体性能正则化奖励学习，显著提升任务成功率与回合回报。
摘要：
为连续控制机器人设计奖励函数往往会引发细微的对齐偏差或奖励黑客问题，在复杂任务中尤其如此。基于偏好的强化学习通过从比较反馈而非手工设计的信号中学习奖励，部分缓解了这些缺陷，但扩展人类标注规模仍具挑战。近期研究利用视觉语言模型（Vision-Language Models，VLM）实现偏好标注自动化，但单张最终状态图像通常无法捕捉智能体的完整运动。\n本文提出一种由两部分组成的解决方案，既提升反馈准确性，又使奖励学习与智能体策略更好地对齐。第一，我们在最终观测图像上叠加轨迹示意图以展示所走路径，让VLM能提供更可靠的偏好——在metaworld任务中将偏好准确率提升约15-20%。第二，我们通过纳入智能体的性能对奖励学习进行正则化，确保奖励模型基于当前策略生成的数据进行优化；这一改进在 locomotion 任务中将回合回报提升了20-30%。在metaworld上的实证研究表明，我们的方法在所有任务中均实现了约70-80%的成功率，而标准方法的成功率低于50%。这些结果凸显了将更丰富的视觉表示与感知智能体的奖励正则化相结合的有效性。
英文摘要：
Designing reward functions for continuous-control robotics often leads to subtle misalignments or reward hacking, especially in complex tasks. Preference-based RL mitigates some of these pitfalls by learning rewards from comparative feedback rather than hand-crafted signals, yet scaling human annotations remains challenging. Recent work uses Vision-Language Models (VLMs) to automate preference labeling, but a single final-state image generally fails to capture the agent's full motion. In this paper, we present a two-part solution that both improves feedback accuracy and better aligns reward learning with the agent's policy. First, we overlay trajectory sketches on final observations to reveal the path taken, allowing VLMs to provide more reliable preferences-improving preference accuracy by approximately 15-20% in metaworld tasks. Second, we regularize reward learning by incorporating the agent's performance, ensuring that the reward model is optimized based on data generated by the current policy; this addition boosts episode returns by 20-30% in locomotion tasks. Empirical studies on metaworld demonstrate that our method achieves, for instance, around 70-80% success rate in all tasks, compared to below 50% for standard approaches. These results underscore the efficacy of combining richer visual representations with agent-aware reward regularization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
