arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.14249cs.CL
收藏

哪种推理轨迹能更好地教会学生推理?一个信息对齐的简单度量

Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);University of Toronto(多伦多大学);University of Sydney(悉尼大学)
Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, … 展开作者

Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

提出Rank-Surprisal Ratio (RSR)度量,通过结合对齐性和信息性评估推理轨迹对学生模型的适用性,在轨迹选择和教师选择中显著优于现有方法。

摘要:

长链思维(CoT)轨迹为从教师到学生大语言模型的推理蒸馏提供了丰富的监督信号。然而,先前工作和我们的实验均表明,来自更强教师的轨迹并不一定能产生更好的学生,凸显了蒸馏中数据-学生适配性的重要性。现有方法主要通过学生似然评估适配性,倾向于选择与学生模型当前行为高度一致的轨迹,但忽略了更具信息性的轨迹。针对这一问题,我们提出Rank-Surprisal Ratio (RSR),一个简单的度量,同时捕捉对齐性和信息性以评估推理轨迹的适用性。RSR的动机源于有效轨迹通常通过结合低绝对概率和相对高排名的token(在学生模型下)来平衡学习信号强度和行为对齐。具体而言,RSR定义为轨迹的平均token级排名与其平均负对数似然之比,计算和解释直观。在五个学生模型和来自11个不同教师的推理轨迹上,RSR与训练后推理性能强相关(平均Spearman 0.86),持续优于现有度量。我们进一步展示了其在轨迹选择和教师选择中的实际效用。

英文摘要:

Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. RSR is motivated by the observation that effective trajectories typically balance learning signal strength and behavioral alignment by combining low absolute probability with relatively high-ranked tokens under the student model. Concretely, RSR is defined as the ratio of a trajectory's average token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training reasoning performance (average Spearman 0.86), consistently outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑