arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05765cs.AI
收藏

RL-VLA$^3$:一种灵活且异步的强化学习框架用于VLA训练

RL-VLA$^3$: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training

Peking University(北京大学);Tsinghua University(清华大学);Tianjin University(天津大学);JDT AI Infra(京东科技AI基础设施);Swinburne University of Technology(斯威本科技大学)
Haoran Sun, Yongjian Guo, Zhong Guan, Shuai Di, Xiaodong Bai, Jing Long, Tianyun Zhao, Mingxi Luo, Hongke Zhao, Likang Wu, Xiaotie Deng, Xu Chu, Xi Xiao, Sheng … 展开作者

Haoran Sun, Yongjian Guo, Zhong Guan, Shuai Di, Xiaodong Bai, Jing Long, Tianyun Zhao, Mingxi Luo, Hongke Zhao, Likang Wu, Xiaotie Deng, Xu Chu, Xi Xiao, Sheng Wen, Yicheng Gong, Junwu Xiong

AI总结:

本文提出RL-VLA$^3$,一种异步强化学习框架,通过动态批调度和灵活环境分片策略,提升VLA训练效率,实验显示其在8至256GPU上均取得85.2%的吞吐量提升。

摘要:

强化学习(RL)已成为训练视觉-语言-动作(VLA)模型的关键范式,使具身智能体通过环境交互进行适应和改进。然而,现有RL框架在VLA训练中继承了传统LLM训练的同步设计原则,将整个rollouts视为不可分割的单元,并严格交替数据收集与策略优化。这与VLA训练的独特特性不匹配,因为物理模拟器引入了高变异性且资源密集的延迟。为此,我们引入RL-VLA$^3$,一种完全异步的分布式RL框架,通过动态批调度器和灵活的环境分片策略,实现模拟、推理和训练组件之间的细粒度异步交互。在多样化的模拟后端、VLA架构和RL算法上的广泛实验表明,RL-VLA$^3$在同步基线上实现了高达85.2%的吞吐量提升,同时保持相同的样本效率,其可扩展性已从8到256个GPU验证。据我们所知,RL-VLA$^3$是首个专门针对VLA训练系统级挑战的完全异步RL训练框架。

英文摘要:

Reinforcement learning (RL) has emerged as a critical paradigm for post-training Vision-Language-Action (VLA) models, enabling embodied agents to adapt and improve through environmental interaction. However, existing RL frameworks for VLAs inherit synchronous design principles from traditional LLM training, treating entire rollouts as indivisible units and alternating strictly between data collection and policy optimization. This fundamentally mismatches the unique characteristics of VLA training, as physical simulators introduce highly variable, resource-intensive latencies. To address this, we introduce RL-VLA$^3$, a fully asynchronous distributed RL framework that enables fine-grained asynchronous interaction between simulation, inference, and training components through dynamic batching schedulers and flexible environment sharding strategies. Extensive experiments across diverse simulation backends, VLA architectures, and RL algorithms demonstrate that RL-VLA$^3$ achieves throughput improvements of up to 85.2\% over synchronous baselines while maintaining identical sample efficiency, with scalability validated from 8 to 256 GPUs. To our knowledge, RL-VLA$^3$ is the first fully asynchronous RL training framework tailored specifically for the system-level challenges of VLA training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑