逐步信息性假设:为什么在大语言模型中熵动态与推理相关?
The Stepwise Informativeness Assumption: When Do Entropy Dynamics Track Reasoning Correctness in LLMs?
AI总结:
本文探讨了大语言模型中熵动态与推理正确性相关的原因,提出逐步信息性假设,并通过实验验证其在多个基准和模型中的有效性。
摘要:
最近的研究使用基于熵的信号在多个表示层次上研究大语言模型的推理能力,但该领域仍主要依赖经验方法。一个核心未解的问题是为什么内部熵动态,即在模型预测分布下定义的熵动态,与外部正确性(由真实答案给出)有如此强大的相关性。本文认为这种相关性源于自回归模型在生成过程中通过答案信息性前缀积累关于真实答案的信息而正确推理。我们通过逐步信息性假设(SIA)正式化这一直觉,该假设指出随着生成过程的推进,推理前缀在期望上积累与答案相关的信息。我们证明SIA自然地从人类推理轨迹上的最大似然优化中出现,并通过标准微调和强化学习流程得到加强。然后,我们推导出SIA的可观察特征,将条件答案熵动态与正确性联系起来。我们通过多个推理基准(GSM8K、ARC、SVAMP)和一组多样化的开源大语言模型(Gemma-2、LLaMA-3.2、Qwen-2.5、DeepSeek和Olmo变体)经验性地测试SIA,显示训练诱导了SIA,并且正确轨迹表现出特征性的条件答案熵模式。
英文摘要:
Recent work uses entropy-based signals at multiple representation levels to study reasoning in large language models, but the field remains largely empirical. From a theoretical perspective, reductions in model uncertainty do not necessarily reflect progress toward correctness. In this work, we characterize when such alignment can be expected through the Stepwise Informativeness Assumption (SIA), which requires entropy-reducing reasoning-prefix extensions increase the probability assigned to valid answers in expectation. We show that next-token prediction can transfer answer-relevant information from training data to the model, but does not, by itself, guarantee alignment with correctness. Instead, we propose that this alignment depends on how the training regime couples reasoning traces to correct answers. Furthermore, we hypothesize observable signatures of this phenomenon and characterize its expected failure regimes. We empirically investigate how entropy dynamics relate to correctness across four reasoning benchmarks (GSM8K, ARC, SVAMP, MATH) and diverse open-weight language models spanning base, instruction-tuned, and RL-trained regimes. Our results show that entropy-correctness alignment strengthens with post-training but can weaken when task difficulty exceeds model capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。