可泛化的密集奖励用于长周期机器人任务
Generalizable Dense Reward for Long-Horizon Robotic Tasks
AI总结：
本文提出VLLR框架，结合外部奖励和内部奖励提升长周期机器人任务性能，通过任务分解和自信心引导实现无需人工奖励工程的高效训练。
摘要：
现有机器人基础策略主要通过大规模模仿学习训练。尽管这些模型表现强劲，但它们在长周期任务中常因分布偏移和误差累积而表现不佳。强化学习（RL）可以微调这些模型，但无法在不同任务间有效工作，除非进行人工奖励工程。我们提出了VLLR，一种结合（1）来自大型语言模型（LLMs）和视觉-语言模型（VLMs）的外部奖励以识别任务进展，以及（2）基于策略自信心的内在奖励的密集奖励框架。VLLR利用LLMs将任务分解为可验证的子任务，然后使用VLMs估计进展以初始化价值函数，避免在完整训练期间的高昂推理成本；自信心提供PPO微调过程中的每一步内在指导。消融研究揭示了互补益处：基于VLM的价值初始化主要提高任务完成效率，而自信心主要增强成功率，特别是在分布外任务上。在涵盖移动操作和导航的CHORES基准上，VLLR在预训练策略上实现了高达56%的绝对成功率提升，在分布内任务上比最先进的RL微调方法高出5%，在分布外任务上高出10%，且无需人工奖励工程。额外的可视化可在https://silongyong.github.io/vllr_project_page/找到。
英文摘要：
Existing robotic foundation policies are trained primarily via large-scale imitation learning. While such models demonstrate strong capabilities, they often struggle with long-horizon tasks due to distribution shift and error accumulation. While reinforcement learning (RL) can finetune these models, it cannot work well across diverse tasks without manual reward engineering. We propose VLLR, a dense reward framework combining (1) an extrinsic reward from Large Language Models (LLMs) and Vision-Language Models (VLMs) for task progress recognition, and (2) an intrinsic reward based on policy self-certainty. VLLR uses LLMs to decompose tasks into verifiable subtasks and then VLMs to estimate progress to initialize the value function for a brief warm-up phase, avoiding prohibitive inference cost during full training; and self-certainty provides per-step intrinsic guidance throughout PPO finetuning. Ablation studies reveal complementary benefits: VLM-based value initialization primarily improves task completion efficiency, while self-certainty primarily enhances success rates, particularly on out-of-distribution tasks. On the CHORES benchmark covering mobile manipulation and navigation, VLLR achieves up to 56% absolute success rate gains over the pretrained policy, up to 5% gains over state-of-the-art RL finetuning methods on in-distribution tasks, and up to $10\%$ gains on out-of-distribution tasks, all without manual reward engineering. Additional visualizations can be found in https://silongyong.github.io/vllr_project_page/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
