arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04992cs.CL
收藏

从错误中学习:负推理样本增强领域外泛化

Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization

CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China(中国科学院人工智能安全重点实验室,计算技术研究所,中国科学院,北京,中国);Harbin Institute of Technology, Harbin, China(哈尔滨工业大学,哈尔滨,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Tsinghua University, Beijing, China(清华大学,北京,中国)

Xueyun Tian, Minghua Ma, Bingbing Xu, Nuoyan Lyu, Wei Li, Heng Dong, Zheng Chu, Yuanzhuo Wang, Huawei Shen

AI总结:

通过引入负例推理样本,GLOW方法有效提升了大语言模型在领域外任务中的泛化能力,通过调节损失下降和提升策略熵来减少过拟合并促进探索。

摘要:

监督微调(SFT)在思维链(CoT)轨迹演示上是一种常见的使大语言模型具备推理能力的方法。常规做法通常只保留最终答案正确的轨迹(正例),而忽略其余轨迹(负例)。我们认为这种范式丢弃了大量监督信息,并加剧了过拟合,限制了领域外(OOD)泛化能力。具体来说,我们意外发现将负例轨迹纳入SFT训练能显著提升OOD泛化能力,因为这些轨迹往往保留了有效的中间推理,尽管最终答案错误。为了深入理解这种影响,我们系统地分析了数据、训练动态和推理行为,识别出22种在负例链中反复出现的模式,这些模式具有双重作用:它们调节损失下降以缓解训练中的过拟合,并在推理中通过提升策略熵35.67%来促进探索。受这些观察的启发,我们进一步提出了基于收益的损失加权(GLOW),一种自适应的、样本感知的方案,通过重新缩放每个样本的损失来利用这种独特的训练动态。实证表明,GLOW高效地利用未过滤的轨迹,在Qwen2.5-7B上比正例-only SFT提升了5.51%的OOD性能,并将MMLU从72.82%提升到76.47%作为强化学习初始化。

英文摘要:

Supervised fine-tuning (SFT) on chain-of-thought (CoT) trajectories demonstrations is a common approach for enabling reasoning in large language models. Standard practices typically only retain trajectories with correct final answers (positives) while ignoring the rest (negatives). We argue that this paradigm discards substantial supervision and exacerbates overfitting, limiting out-of-domain (OOD) generalization. Specifically, we surprisingly find that incorporating negative trajectories into SFT yields substantial OOD generalization gains over positive-only training, as these trajectories often retain valid intermediate reasoning despite incorrect final answers. To understand this effect in depth, we systematically analyze data, training dynamics, and inference behavior, identifying 22 recurring patterns in negative chains that serve a dual role: they moderate loss descent to mitigate overfitting during training and boost policy entropy by 35.67% during inference to facilitate exploration. Motivated by these observations, we further propose Gain-based LOss Weighting (GLOW), an adaptive, sample-aware scheme that exploits such distinctive training dynamics by rescaling per-sample loss based on inter-epoch progress. Empirically, GLOW efficiently leverages unfiltered trajectories, yielding a 5.51% OOD gain over positive-only SFT on Qwen2.5-7B and boosting MMLU from 72.82% to 76.47% as an RL initialization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑