一种用于机器人真实世界强化学习的视觉-语言-动作-评判模型
A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
AI总结：
本文提出VLAC模型，将过程奖励评判与动作策略统一于单一视觉-语言模型，通过密集进度信号、一次性迁移和人在环分级干预，在真实机器人操作任务中显著提升样本效率与成功率。
摘要：
基于视觉-语言-动作（VLA）模型的机器人真实世界强化学习（RL）受到稀疏、手工设计奖励和低效探索的瓶颈制约。我们提出VLAC，这是一种通用过程奖励模型，构建于InternVL之上，并在大规模异构数据集上进行训练。给定成对观测和一个语言目标，它输出密集的进度增量和完成信号，从而消除了针对特定任务的奖励工程，并支持对未见任务和环境的一次性上下文迁移。VLAC在视觉-语言数据集上进行训练，以增强感知、对话和推理能力，同时结合机器人和人类轨迹数据来支撑动作生成和进度估计，并通过构建大量负样本和语义不匹配样本来进一步增强其拒绝无关提示以及检测倒退或停滞的能力。通过提示控制，单个VLAC模型交替生成奖励和动作令牌，统一了评判器与策略。部署在异步真实世界RL循环中，我们叠加了一个分级的人在环协议（离线演示回放、返回并探索、人类引导探索），以加速探索并稳定早期学习。在四个不同的真实世界操作任务中，VLAC在200个真实世界交互回合内将成功率从约30%提升到约90%；结合人在环干预后，样本效率进一步提高了50%，最终成功率最高达到100%。
英文摘要：
Robotic real-world reinforcement learning (RL) with vision-language-action (VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient exploration. We introduce VLAC, a general process reward model built upon InternVL and trained on large scale heterogeneous datasets. Given pairwise observations and a language goal, it outputs dense progress delta and done signal, eliminating task-specific reward engineering, and supports one-shot in-context transfer to unseen tasks and environments. VLAC is trained on vision-language datasets to strengthen perception, dialogic and reasoning capabilities, together with robot and human trajectories data that ground action generation and progress estimation, and additionally strengthened to reject irrelevant prompts as well as detect regression or stagnation by constructing large numbers of negative and semantically mismatched samples. With prompt control, a single VLAC model alternately generating reward and action tokens, unifying critic and policy. Deployed inside an asynchronous real-world RL loop, we layer a graded human-in-the-loop protocol (offline demonstration replay, return and explore, human guided explore) that accelerates exploration and stabilizes early learning. Across four distinct real-world manipulation tasks, VLAC lifts success rates from about 30\% to about 90\% within 200 real-world interaction episodes; incorporating human-in-the-loop interventions yields a further 50% improvement in sample efficiency and achieves up to 100% final success.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
