arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.23740cs.LGcs.AI
收藏

离线推理训练的权重空间几何

Weight-Space Geometry of Offline Reasoning Training

Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

AI总结:

通过余弦相似度、主角度子空间分析等方法,研究六种离线强化学习方法在数学推理任务中的权重更新几何,发现SFT、RFT、RIFT几乎共线,DFT方向偏离,Offline GRPO添加正交分量,DPO位于近正交子空间且准确率最高。

摘要:

离线强化学习损失(RFT、RIFT、DFT、Offline GRPO、DPO)被广泛用于将大型教师模型的推理能力蒸馏到较小的学生模型中,并且通常仅通过下游准确率进行比较。我们询问这些方法在机制上是不同的还是收敛到相似的权重更新。在相同的数学 rollout 数据上,从单个基础模型(Qwen3-4B)出发,使用仅注意力的 LoRA 训练六种方法(SFT、RFT、DFT、RIFT、Offline GRPO、DPO),我们通过余弦相似度、主角度子空间分析、线性模式连通性和 CKA 分析结果中的权重增量。我们观察到:(i)SFT、RFT 和 RIFT 的权重增量几乎共线(余弦 >= 0.97,144个模块的中位 top-1 主角度约7度),并且 GSM8K 准确率相当(87-88%,n=1319;成对 McNemar p >= 0.15);(ii)DFT 在方向上比任何奖励加权方法偏离更远,尽管使用相同的数据;(iii)Offline GRPO 添加了与 SFT 方向正交的显著分量(全局约67%,后期层高达约86%),同时保持在 SFT 损失盆地中;(iv)DPO 位于近正交子空间,存在模式连通性障碍,并将后期层 CKA 降低到约0.46。DPO 在我们的协议中在 GSM8K(93.5%,与其他方法的成对 McNemar p < 10^-9)和 AIME26(30.0% 对比 3.3-10.0%)上均达到最高准确率;其训练使用的学习率比其他方法小10倍(标准惯例),因此更新范数和准确率差距反映了损失函数和优化器选择的共同影响,学习率匹配的 DPO 比较留待未来工作。

英文摘要:

Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone. We ask whether they are mechanistically distinct or converge to a similar weight update. Training six methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical math rollouts from a single base model (Qwen3-4B) with attention-only LoRA, we analyze the resulting deltas via cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA. We observe: (i) SFT, RFT, and RIFT have nearly colinear weight deltas (cosine >= 0.97, top-1 principal angle ~7 deg median over 144 modules) and comparable GSM8K accuracy (87-88%, n=1319; pairwise McNemar p >= 0.15); (ii) DFT diverges further in direction than any reward-weighted method despite using the same data; (iii) Offline GRPO adds a substantial component orthogonal to the SFT direction (~67% globally, up to ~86% in late layers) while staying in the SFT loss basin; (iv) DPO sits in a near-orthogonal subspace, shows a mode-connectivity barrier, and collapses late-layer CKA to ~0.46. DPO also reaches the highest accuracy in our protocol on both GSM8K (93.5%, McNemar p < 10^-9 vs. each other method) and AIME26 (30.0% vs. 3.3-10.0%); its training uses a 10x smaller learning rate than the others (the standard convention), so the update-norm and accuracy gaps reflect loss-function and optimizer choices jointly, and a learning-rate-matched DPO comparison is left for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑