在部署中学习：面向通用机器人策略的机群规模强化学习
Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
AI总结：
提出LWD框架，通过机群规模的离线到在线强化学习，结合分布式隐式价值学习与伴随匹配Q学习，持续后训练通用视觉-语言-动作策略，在16台双臂机器人上实现95%平均成功率。
摘要：
通用机器人策略日益受益于大规模预训练，但仅靠离线数据不足以实现稳健的实世界部署。已部署的机器人会遇到分布偏移、长尾故障、任务变化以及人类纠正机会，这些是固定演示数据集无法完全捕获的。我们提出了“在部署中学习”（LWD），一个机群规模的离线到在线强化学习框架，用于通用视觉-语言-动作（VLA）策略的持续后训练。从预训练的VLA策略开始，LWD通过使用在机器人机群中收集的自主 rollout 和人类干预，在部署、共享物理经验、策略改进和重新部署之间形成闭环。为了稳定地从异构、稀疏奖励的机群数据中学习，LWD结合了分布式隐式价值学习（DIVL）进行鲁棒的价值估计，以及通过伴随匹配的Q学习（QAM）在基于流的VLA动作生成器中进行策略提取。我们在一个由16台双臂机器人组成的机群上，在八个真实世界操作任务（包括语义杂货补货和3-5分钟的长时域任务）上验证了LWD。单个通用策略随着机群经验的积累而改进，平均成功率达到95%，在长时域任务上提升最大。
英文摘要：
Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
