通过模块化梯度手术提升通用推理模型
Advancing General-Purpose Reasoning Models with Modular Gradient Surgery
AI总结：
通过模块化梯度手术提升通用推理模型，解决跨领域训练中的梯度冲突问题，提升多任务RL性能。
摘要：
强化学习（RL）在最近大规模推理模型（LRMs）的进展中发挥了核心作用，带来了在可验证和开放性推理方面的显著提升。然而，训练一个跨多个领域的通用LRM仍然具有挑战性，因为领域异质性明显。通过系统研究两种广泛使用的策略，顺序RL和混合RL，我们发现这两种策略在行为和梯度层面都产生了显著的跨领域干扰，导致总体收益有限。为了解决这些挑战，我们引入了**M**odular **G**radient **S**urgery（**MGS**），它在transformer内部在模块层面解决梯度冲突。当应用于Llama和Qwen模型时，MGS在三个代表性领域（数学、一般聊天和指令遵循）上的标准多任务RL中分别实现了平均改进4.3（16.6%）和4.5（11.1%）的提升。进一步分析显示，MGS在长时间训练下仍保持有效。总体而言，我们的研究澄清了多领域RL中的干扰源，并为训练通用LRM提供了有效的解决方案。
英文摘要：
Reinforcement learning (RL) has played a central role in recent advances in large reasoning models (LRMs), yielding strong gains in verifiable and open-ended reasoning. However, training a single general-purpose LRM across diverse domains remains challenging due to pronounced domain heterogeneity. Through a systematic study of two widely used strategies, Sequential RL and Mixed RL, we find that both incur substantial cross-domain interference at the behavioral and gradient levels, resulting in limited overall gains. To address these challenges, we introduce **M**odular **G**radient **S**urgery (**MGS**), which resolves gradient conflicts at the module level within the transformer. When applied to Llama and Qwen models, MGS achieves average improvements of 4.3 (16.6\%) and 4.5 (11.1\%) points, respectively, over standard multi-task RL across three representative domains (math, general chat, and instruction following). Further analysis demonstrates that MGS remains effective under prolonged training. Overall, our study clarifies the sources of interference in multi-domain RL and presents an effective solution for training general-purpose LRMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
