语言统计中的对称性塑造了模型表示的几何结构
Symmetry in language statistics shapes the geometry of model representations
AI总结：
本文证明语言统计中的平移对称性（如月份共现频率仅取决于时间间隔）决定了词嵌入模型的高维几何结构，并推导出表示流形的解析形式，该几何在统计扰动下仍保持鲁棒。
摘要：
语言模型学习到的内部表示始终展现出显著的几何结构：日历月份排列成圆形，历史年份形成平滑的一维流形，城市的纬度和经度可以通过线性探针解码。为了解释这种神经编码，我们首先证明语言统计具有平移对称性（例如，任意两个月份在文本中共现的频率仅取决于它们之间的时间间隔）。我们证明这种对称性支配了高维词嵌入模型中的这些几何结构，并解析推导出词表示的流形几何。这些预测在经验上与大型文本嵌入模型和大型语言模型相匹配。此外，当相关统计量受到扰动时（例如，删除所有两个月份共现的句子），表示几何在中等嵌入维度下仍然存在。我们证明，当共现统计量由潜在变量控制时，这种鲁棒性自然出现。我们的结果表明，这些表示流形源于自然语言的统计对称性。
英文摘要：
The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
