arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.03066cs.CLcs.AIcs.LG
收藏

大型语言模型是否编码了推理标记的功能重要性?

Do LLMs Encode Functional Importance of Reasoning Tokens?

Grainger College of Engineering(格雷格纳工程学院);University of Illinois Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校)

Janvijay Singh, Dilek Hakkani-Tür

AI总结:

本文研究了大型语言模型是否内部编码了推理标记的功能重要性,提出贪心剪枝方法,在保持模型似然的前提下剪除对答案生成影响最小的推理标记,验证了模型在压缩推理链时的功能重要性结构。

摘要:

大型语言模型通过生成长推理链解决复杂任务,但会以增加计算成本和降低功能相关性隔离能力为代价,获得更高准确性。先前工作通过概率采样、启发式或前沿模型监督来缩短此类链,但未能明确揭示模型是否在内部编码标记级功能重要性。本文通过诊断方法填补这一空白,提出贪心剪枝,一种在指定目标下迭代删除对模型似然影响最小的推理标记的删除程序,从而生成长度可控的推理链。我们在蒸馏框架下评估了剪枝推理,并证明在匹配推理长度下,学生模型在剪枝链上表现优于前沿模型监督压缩基线。最后,我们的分析揭示了系统性的剪枝模式,并表明注意力分数可以预测贪心剪枝排名,进一步表明模型在推理标记上编码了非平凡的功能重要性结构。

英文摘要:

Large language models solve complex tasks by generating long reasoning chains, achieving higher accuracy at the cost of increased computational cost and reduced ability to isolate functionally relevant reasoning. Prior work on compact reasoning shortens such chains through probabilistic sampling, heuristics, or supervision from frontier models, but offers limited insight into whether models internally encode token-level functional importance for answer generation. We address this gap diagnostically and propose greedy pruning, a likelihood-preserving deletion procedure that iteratively removes reasoning tokens whose removal minimally degrades model likelihood under a specified objective, yielding length-controlled reasoning chains. We evaluate pruned reasoning in a distillation framework and show that students trained on pruned chains outperform a frontier-model-supervised compression baseline at matched reasoning lengths. Finally, our analysis reveals systematic pruning patterns and shows that attention scores can predict greedy pruning ranks, further suggesting that models encode a nontrivial functional importance structure over reasoning tokens.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑