arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.15422cs.LG
收藏

DualKV: 面向高效RL训练的共享提示Flash注意力机制,支持大规模展开和长上下文

DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts

Amazon Web Services(亚马逊网络服务);Google(谷歌);University of Minnesota(明尼苏达大学)

Jiading Gai, Shuai Zhang, Xiang Song, Bernie Wang, George Karypis

AI总结:

针对RL训练中共享提示重复计算问题,提出DualKV内核,通过融合CUDA前向/反向核和veRL数据流水线重排,消除提示复制,实现1.63-3.82倍策略更新加速。

摘要:

现代RL后训练方法(如GRPO和DAPO)在从共享提示($P$个token)采样的$N$个响应序列(每个$R$个token)上进行训练,但标准FlashAttention在前向和反向传播中将所有$P$个提示token复制$N$次——在相同的隐藏状态上重复计算和内存。在大规模展开、长上下文RL训练($N\geq16$,$P\geq8\text{K}$)中,这种冗余主导了策略更新成本。我们观察到,在仅解码器模型中,因果掩码使提示表示在每一层跨序列不变,因此所有逐token操作(归一化、投影、MLP)和注意力可以一次性处理提示——这一特性尚未在训练的内核级别被利用。我们提出\textbf{DualKV},这是首个消除RL训练中共享提示复制的FlashAttention内核变体,通过(1)~融合的CUDA前向和反向内核,在单次内核启动中迭代两个不相交的KV区域——共享上下文和逐序列响应,以及(2)~veRL中的数据流水线重设计,将$N(P{+}R)$个token重新打包为每个微批$P{+}NR$个token,将token减少从注意力扩展到整个模型,因子$ρ= N(P{+}R)/(P{+}NR)$。DualKV在数学上等价于标准注意力,且不引入近似。在Qwen3-8B GRPO训练中,使用8$\times$H100 GPU($N{=}32$,8K上下文),DualKV实现了$1.63$--$2.09\times$的策略更新加速,支持$2\times$更大的微批,并将MFU从$36\%$提升至$76\%$。类似增益在DAPO上成立($2.47\times$加速,$77\%$ MFU)。在30B MoE规模下,使用16$\times$H100,DualKV相比FlashAttention(需要4路Ulysses序列并行以避免OOM)实现了$3.82\times$的策略更新加速和$3.38\times$的端到端步骤加速。

英文摘要:

Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states. In large-rollout, long-context RL training (N>=16, P>=8K), this redundancy dominates the policy update cost. We observe that in decoder-only models, causal masking makes prompt representations invariant across sequences at every layer, so all per-token operations (norms, projections, MLP) and attention can process the prompt once -- a property not yet exploited at the kernel level for training. We propose DualKV, the first FlashAttention kernel variant that eliminates shared-prompt replication during RL training, via (1) fused CUDA forward and backward kernels that iterate over two disjoint KV regions -- shared context and per-sequence response -- in a single kernel launch, and (2) a data-pipeline redesign in veRL that repacks N(P+R) tokens into P+NR tokens per micro-batch, extending the token reduction from attention to the entire model by a factor rho = N(P+R)/(P+NR). DualKV is mathematically equivalent to standard attention and introduces no approximation. On Qwen3-8B GRPO training with 8xH100 GPUs (N=32, 8K-context), DualKV achieves 1.63--2.09x policy-update speedup, enables 2x larger micro-batches, and raises MFU from 36% to 76%. Similar gains hold for DAPO (2.47x speedup, 77% MFU). At 30B MoE scale on 16xH100, DualKV achieves 3.82x policy-update and 3.38x end-to-end step speedup over FlashAttention (which requires 4-way Ulysses sequence parallelism to avoid OOM). DualKV also extends to hybrid sliding/global attention with head dimension 512 (which FA2 does not support) and integrates with Ulysses sequence parallelism, demonstrated on Gemma-4-31B GRPO at 64K context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑