arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.24969cond-mat.stat-mechcs.CLphysics.bio-phq-bio.NC
收藏

大语言模型与英语的熵

Large language models and the entropy of English

Joseph Henry Laboratories of Physics(乔赛亚·亨利物理实验室);Princeton Center for Theoretical Science(普林斯顿理论科学中心);Lewis--Sigler Institute for Integrative Genomics(刘易斯-西格尔整合基因组学研究所);Princeton University(普林斯顿大学);Initiative for the Theoretical Sciences(理论科学倡议);The CUNY Graduate Center(纽约市立大学研究生中心)

Colin Scheibner, Lindsay M. Smith, William Bialek

AI总结:

通过大语言模型揭示英语文本中长程结构的熵特性及依赖关系

摘要:

我们利用大语言模型(LLMs)来揭示来自各种来源的英语文本中的长程结构。在许多情况下,条件熵或编码长度至少在 $N\sim 10^4$ 个字符的情况下持续减少,这表明这些距离之间存在直接依赖或相互作用。一个推论是,在这些分离处,字符之间存在小但显著的相关性,我们从数据中独立于模型展示了这一点。编码长度的分布揭示了随着 $N$ 增大,越来越多的字符表现出一种涌现的确定性。在模型训练过程中,我们观察到在长和短的上下文长度上存在不同的动态,这表明长程结构仅是逐渐学习的。我们的结果限制了构建统计物理模型来描述 LLMs 或语言本身的努力。

英文摘要:

We use large language models (LLMs) to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to decrease with context length at least to $N\sim 10^4$ characters, implying that there are direct dependencies or interactions across these distances. A corollary is that there are small but significant correlations between characters at these separations, as we show from the data independent of models. The distribution of code lengths reveals an emergent certainty about an increasing fraction of characters at large $N$. Over the course of model training, we observe different dynamics at long and short context lengths, suggesting that long-ranged structure is learned only gradually. Our results constrain efforts to build statistical physics models of LLMs or language itself.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑