重新审视大语言模型推理中的均匀信息密度假说
Revisiting the Uniform Information Density Hypothesis in LLM Reasoning
AI总结:
本文研究大语言模型推理中均匀信息密度假说的有效性,提出新框架量化局部和全局信息流均匀性,发现高质量推理在局部均匀但全局非均匀,证明均匀性优于其他内部信号预测推理质量。
摘要:
均匀信息密度(UID)假说提出通过维持信息流的稳定流动实现有效沟通。本文重新审视该原则在大语言模型(LLM)推理中的应用,探讨步骤层面的均匀性是否反映推理质量。为此,我们引入一种新的框架,利用基于熵的逐步密度度量,在局部和全局层面量化信息流的均匀性。在七个推理基准测试中,我们观察到一种反直觉模式:高质量推理表现出局部平滑的逐步过渡和结构化的非均匀信息流,而在轨迹层面呈现全局非均匀性。结果表明,这些均匀性优于其他内部信号作为推理质量预测器,这种与人类沟通的分歧并非模型缺陷,而是人类沟通与LLM推理目标不同的产物。
英文摘要:
The Uniform Information Density (UID) hypothesis proposes that effective communication is achieved by maintaining a stable flow of information. In this work, we revisit this principle in the context of Large Language Model (LLM) reasoning, asking whether step-level uniformity reflects reasoning quality. To this end, we introduce a novel framework to quantify uniformity of information flow at both local and global levels, using an entropy-based stepwise density metric. Across experiments on seven reasoning benchmarks, we see a counter-intuitive pattern: while high-quality reasoning exhibit smooth step-by-step transitions local uniformity and structured, non-uniform information flow at the trajectory level global non-uniformity. The results demonstrate that these uniformities outperform alternative internal signals as predictors of reasoning quality, and such divergence with human communication is not a model deficiency, but a byproduct of distinct objectives between human communication and LLM reasoning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。