语言模型中信息论压缩与几何压缩的衔接
Bridging Information-Theoretic and Geometric Compression in Language Models
AI总结：
该研究从几何与信息论双视角分析预训练语言模型的压缩机制，验证二者高度相关且高压缩率预示模型快速适配，同时首次评估语言数据的固有维度估计器，明确部分有效估计器。
摘要：
语言模型（LM）若要忠实地建模人类语言，必须将海量、甚至可能是无限的信息压缩到相对较少的维度中。本文提出从几何与信息论两个视角分析（预训练）语言模型中的压缩机制。研究证明这两个视角高度相关，即语言数据的固有几何维度能够预测其在语言模型下的编码长度。进一步研究表明，对一个语言数据集的高压缩率预示着模型对该数据集的快速适配，这证实了压缩语言信息的能力是语言模型取得优异性能的重要组成部分。作为本研究分析的一项实用副产品，本文首次在语言数据上评估了一系列固有维度估计器，结果显示仅有部分估计器能够体现信息论压缩、几何压缩与适配难度之间的关系。
英文摘要：
For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
