arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.12784cs.DC
收藏

通过受滞约束的回放协调实现高效的异步RL后训练

StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

Peking University(北京大学);Shanghai Jiao Tong University(上海交通大学);Tencent(腾讯)

Haoyang Li, Sheng Lin, Fangcheng Fu, Yuming Zhou, Xiaodong Ji, Yanfeng Zhao, Lefeng Wang, Jie Jiang, Bin Cui

AI总结:

StaleFlow通过受滞约束的回放协调,解决RL后训练中数据滞留与偏斜问题,提升系统吞吐量。

摘要:

强化学习(RL)后训练已成为增强现代大模型能力的关键。最近的趋势是开发具有完全解耦架构的RL系统,将三个RL阶段(回放、奖励和训练)解耦到不同的资源上,并异步执行。然而,出现了两个关键的数据层面问题:(1)异步执行导致轨迹(由回放生成的数据)出现数据滞留,因为回放中使用的模型参数可能未更新,这会损害RL收敛;(2)轨迹长度的变化引入严重的数据偏斜,导致工作负载不平衡和系统性能下降。现有系统未能以统一的方式解决这两个问题。严格控制数据滞留的技术往往限制了有效数据偏斜缓解,而激进的数据偏斜缓解技术则会加剧数据滞留。因此,系统被迫在收敛性和性能之间进行权衡。为此,我们提出了StaleFlow,一个RL后训练系统,共同解决数据滞留和偏斜问题。首先,为了控制滞留,StaleFlow引入了一个全局一致性协议,跟踪每个轨迹的完整生命周期并约束滞留。其次,为了缓解偏斜,StaleFlow重新设计RL系统架构,通过构建轨迹和参数的数据服务器来实现灵活的回放协调。随后,我们开发了一套受滞留影响、以吞吐量为导向的策略来提高系统性能。评估显示,StaleFlow在吞吐量方面比最先进的系统高1.42-2.68倍(1.17-2.01倍平均),而不会影响收敛。

英文摘要:

Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregated architecture, which decouples the three RL phases (rollout, reward, and training) onto separate resources and executes them asynchronously. However, two critical data-level concerns arise: (1) asynchronous execution leads to data staleness in trajectories (the data generated by rollout) as the model parameters used in rollout may not be up to date, which impairs RL convergence; and (2) the length variation of trajectories introduces severe data skewness, leading to workload imbalance and degraded system performance. Existing systems fail to address these two concerns in a unified manner. Techniques that tightly control data staleness often constrain effective data skewness mitigation, while aggressive data skewness mitigation tends to exacerbate data staleness. As a result, systems are forced to trade off convergence for performance, or vice versa. To address this, we propose StaleFlow, an RL post-training system that jointly tackles data staleness and skewness. First, to control staleness, StaleFlow introduces a global consistency protocol that tracks the full lifecycle of each trajectory and constrains staleness. Second, to mitigate skewness, StaleFlow re-designs the RL system architecture by constructing data servers for trajectories and parameters to achieve flexible rollout coordination. Subsequently, we develop a suite of staleness-aware, throughput-oriented strategies to enhance system performance. Evaluations show that StaleFlow achieves up to 1.42-2.68$\times$ (1.18-1.91$\times$ on average) higher throughput than state-of-the-art systems, without compromising convergence. Our source code is available: https://github.com/psrl-project/psrl.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑