arXiv 2505.187192025-05-24cs.ROcs.AI

VLA-RL：迈向具有可扩展强化学习的高超通用机器人操作

VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）

Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, Ziwei Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出VLA-RL框架，利用在线强化学习改进预训练自回归VLA模型，通过轨迹级RL公式和过程奖励模型，在LIBERO任务上超越最强基线4.5%，并匹配商业模型性能，展现推理缩放潜力。

摘要：

近期，高容量视觉-语言-动作（VLA）模型通过模仿人类示范，在一系列机器人操作任务中展现了令人印象深刻的性能。然而，利用有限访问状态的离线数据会在分布外场景中导致执行失败。直观上，一种基于探索的方法在测试时改进在线收集的数据可以解决这一局限性。我们提出了VLA-RL，一个算法和系统框架，利用在线强化学习（RL）来改进预训练的自回归VLA在下游任务中的表现。在统一视角下，我们首先引入了用于自回归VLA训练的轨迹级RL公式，将一般机器人操作轨迹建模为多模态多轮对话。为了解决稀疏奖励的挑战，我们微调了一个预训练的视觉-语言模型作为机器人过程奖励模型，该模型在自动提取的任务片段上标注的伪奖励标签上进行训练。为了扩展规模，我们识别了若干提高稳定性和效率的实现发现，包括课程选择策略、GPU平衡的向量化环境、批量解码和评论家预热。VLA-RL使OpenVLA-7B在LIBERO的40个具有挑战性的机器人操作任务中超越了最强的微调基线4.5%，甚至达到了先进商业模型（如π0-FAST）的性能。值得注意的是，我们观察到VLA-RL受益于增加的测试时优化，表明机器人学中推理缩放定律的早期火花。

英文摘要：

Recent high-capacity vision-language-action (VLA) models have demonstrated impressive performance on a range of robotic manipulation tasks by imitating human demonstrations. However, exploiting offline data with limited visited states will cause execution failure in out-of-distribution scenarios. Intuitively, an exploration-based method that improves on online collected data at test time could address this limitation. We present VLA-RL, an algorithmic and systematic framework that leverages online reinforcement learning (RL) to improve pretrained auto-regressive VLAs in downstream tasks. Within a unified perspective, we first introduce a trajectory-level RL formulation for auto-regressive VLA training, which models general robotic manipulation trajectory as multi-modal multi-turn conversation. To address the challenge of sparse rewards, we fine-tune a pretrained vision-language model as a robotic process reward model, which is trained on pseudo reward labels annotated on automatically extracted task segments. To scale up, we identify several implementation findings that improve the stability and efficiency including curriculum selection strategy, GPU-balanced vectorized environments, batch decoding, and critic warmup. VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO, and even matches the performance of advanced commercial models such as $π_0$-FAST. Notably, we observe that VLA-RL benefits from increased test-time optimization, indicating an early spark of inference scaling laws in robotics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。