QaRL：基于回放对齐的量化感知强化学习用于在训练-推理不匹配下的快速稳定训练
QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
AI总结：
本文提出QaRL，通过对齐训练和回放的量化过程，解决训练-推理不匹配问题，提升训练效率和稳定性，同时保持低比特吞吐量优势。
摘要：
大型语言模型（LLM）的强化学习（RL）流水线常因回放生成而成为瓶颈，导致端到端训练缓慢。近期工作通过在回放中运行量化来加速解码，这是RL循环中最昂贵的阶段。然而，这些设置通过放大训练-推理差距而破坏了优化：回放在低精度下运行，而学习更新在全精度下计算。为解决这一挑战，我们提出了QaRL（回放对齐量化感知强化学习），通过对齐训练端的前向计算与量化回放以最小化不匹配。我们进一步识别出量化回放中的失败模式：长形式响应倾向于产生重复、乱码的token（错误token）。为缓解这些问题，我们引入TBPO（信任带策略优化），一种序列级目标，具有双重剪裁用于负样本，旨在保持更新在信任区域内。在Qwen3-30B-A3B MoE用于数学问题上，QaRL在量化回放训练上表现更优，提升5.5%，同时提高稳定性和保留低比特吞吐量优势。
英文摘要：
Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
