arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.11554cs.CL
收藏

Relax:一种用于大规模多模态后训练的异步强化学习引擎

Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

AI Platform, Xiaohongshu Inc(小红书AI平台);The University of Hong Kong(香港大学);University of Science and Technology of China(中国科学技术大学)

Liujie Zhang, Benzhe Ning, Rui Yang, Xiaoyan Yu, Jiaxing Li, Lumeng Wu, Jia Liu, Minghao Li, Weihang Chen, Weiqi Hu, Lei Zhang

AI总结:

Relax通过三个协同设计的架构层解决多模态数据流、大规模鲁棒性和延迟-吞吐量平衡问题,实现比veRL快1.20倍的端到端加速,并在多模态强化学习中表现出稳定收敛性。

摘要:

强化学习(RL)后训练已被证明在大型语言模型中有效,解锁推理、自我反思和工具使用能力。随着模型扩展到多模态输入和代理多轮工作流,RL训练系统面临三个相互依赖的挑战:异构数据流、大规模操作鲁棒性和延迟-吞吐量权衡。我们提出了Relax(Reinforcement Engine Leveraging Agentic X-modality),一种开源的RL训练引擎,通过三个协同设计的架构层解决这些挑战。首先,一个多模态原生架构将多模态支持融入整个栈——从数据预处理和模态感知并行到推理生成——而不是将其 retrofit 到以文本为中心的管道中。其次,每个RL角色都作为独立、故障隔离的服务运行,可以扩展、恢复和升级,而无需全局协调。第三,服务级解耦通过TransferQueue数据总线实现异步训练,其中单个延迟参数平滑地在策略内、近策略内和完全异步执行之间插值。Relax在Qwen3-4B的策略内训练中比veRL快1.20倍。其完全异步模式在Qwen3-4B上比colocate快1.76倍,在Qwen3-Omni-30B上快2.00倍,所有模式都收敛到相同的奖励水平。Relax在MoE模型中仅以1.9%的开销支持R3(Rollout Routing Replay)~\cite{ma2025r3},而veRL在同一配置下会降级32%。它进一步在Qwen3-Omni上展示了稳定的多模态RL收敛性,持续超过2,000步视频数据而不降级。Relax可在https://github.com/rednote-ai/Relax上获得。

英文摘要:

Reinforcement learning (RL) post-training has proven effective at unlocking reasoning, self-reflection, and tool-use capabilities in large language models. As models extend to omni-modal inputs and agentic multi-turn workflows, RL training systems face three interdependent challenges: heterogeneous data flows, operational robustness at scale, and the staleness -- throughput tradeoff. We present \textbf{Relax} (Reinforcement Engine Leveraging Agentic X-modality), an open-source RL training engine that addresses these challenges through three co-designed architectural layers. First, an \emph{omni-native architecture} builds multimodal support into the full stack -- from data preprocessing and modality-aware parallelism to inference generation -- rather than retrofitting it onto a text-centric pipeline. Second, each RL role runs as an independent, fault-isolated service that can be scaled, recovered, and upgraded without global coordination. Third, service-level decoupling enables asynchronous training via the TransferQueue data bus, where a single staleness parameter smoothly interpolates among on-policy, near-on-policy, and fully asynchronous execution. Relax achieves a 1.20$\times$ end-to-end speedup over veRL on Qwen3-4B on-policy training. Its fully async mode delivers a 1.76$\times$ speedup over colocate on Qwen3-4B and a 2.00$\times$ speedup on Qwen3-Omni-30B, while all modes converge to the same reward level. Relax supports R3 (Rollout Routing Replay)~\cite{ma2025r3} for MoE models with only 1.9\% overhead, compared to 32\% degradation in veRL under the same configuration. It further demonstrates stable omni-modal RL convergence on Qwen3-Omni across image, text, and audio, sustaining over 2{,}000 steps on video without degradation. Relax is available at https://github.com/rednote-ai/Relax.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑