arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03516cs.LGcs.AI
收藏

并非所有负样本都平等:LLM通过合理推理学习更好

Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning

East China Normal University(华东师范大学);Fudan University(复旦大学);Independent Researcher(独立研究者)

Zixiang Di, Jinyi Han, Shuo Zhang, Ying Liao, Zhi Li, Xiaofeng Ji, Yongqi Wang, Zheming Yang, Ming Gao, Bingdong Li, Jie Wang

AI总结:

本文提出PNS方法,通过合成高质量负样本提升LLM推理能力,实验显示其在多个基准测试中优于其他方法。

摘要:

从负样本学习具有提升大语言模型(LLM)推理能力的巨大潜力,但现有方法将所有错误响应视为等信息量,忽视了样本质量的关键作用。为此,我们提出了 plausible negative samples(PNS),一种合成高质量负样本的方法,这些样本表现出预期的格式和结构连贯性,但最终得出错误答案。PNS通过反向强化学习(RL)训练专用模型,由结合格式合规性、准确性反转、奖励模型评估和思维链评估的复合奖励引导,生成几乎与正确解决方案无法区分的响应。我们进一步验证PNS作为首选优化的数据源,在三个基础模型上七个数学推理基准测试中均有效。结果表明,PNS在其他负样本合成方法上始终表现更优,平均比RL训练模型提高2.03%。

英文摘要:

Learning from negative samples holds great promise for improving Large Language Model (LLM) reasoning capability, yet existing methods treat all incorrect responses as equally informative, overlooking the crucial role of sample quality. To address this, we propose Plausible Negative Samples (PNS), a method that synthesizes high-quality negative samples exhibiting expected format and structural coherence while ultimately yielding incorrect answers. PNS trains a dedicated model via reverse reinforcement learning (RL) guided by a composite reward combining format compliance, accuracy inversion, reward model assessment, and chain-of-thought evaluation, generating responses nearly indistinguishable from correct solutions. We further validate PNS as a plug-and-play data source for preference optimization across three backbone models on seven mathematical reasoning benchmarks. Results demonstrate that PNS consistently outperforms other negative sample synthesis methods, achieving an average improvement of 2.03% over RL-trained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑