发表机构
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文通过对角模型Fisher统一解释了大语言模型后训练中参数更新的稀疏性，证明其可预测梯度集中位置，并揭示高概率词元敏感性相似导致低Fisher的机制。
AI 中文摘要
近期研究观察到，语言模型后训练过程中的参数变化可能集中在一小部分坐标上。这一现象已在强化学习、同策略蒸馏以及在近策略数据上的监督微调中被报道。它在不同后训练范式中的反复出现，暗示了训练动态中存在共享结构。在本文中，我们通过对角模型Fisher来审视这一模式，该Fisher度量模型输出分布对单个参数的敏感性，并且独立于任何特定的奖励或教师信号。理论上，我们证明较小的对角Fisher会导致一系列训练目标下较小的期望梯度，为稀疏梯度更新提供了统一的解释。实证上，我们在强化学习（RL）和同策略蒸馏（OPD）中检验了这一联系。我们发现Fisher能够识别梯度集中的位置，并且从初始Fisher中选择的固定稀疏掩码保留了全量训练改进的很大比例。最后，我们研究了同策略训练中低Fisher的潜在机制。我们的结果表明，高概率的下一个词元往往具有相似的参数敏感性，这导致了低Fisher。综合来看，这些结果确立了将更新稀疏性与大语言模型后训练中的同策略训练动态联系起来的统一视角——对角模型Fisher。
英文摘要
Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training.
