发表机构
Johns Hopkins University; EPFL; SISSA（约翰斯·霍普金斯大学; 洛桑联邦理工学院; 国际高等研究学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出理论框架，基于层次结构假设解释多语言模型中跨语言表示相似性，区分相似性与对齐，并预测通过减去线性可预测分量增强相似性，在预训练LLM中得到验证。
AI 中文摘要
多语言语言模型的内部层中，翻译句子的表示是相似的——这一观察与柏拉图式表示假说相关，但尚未得到理论解释。我们基于一个假设提供解释：数据具有隐藏的层次结构，其抽象层在语言间共享，而表层则特定于模态或语言。具体地，我们从概率上下文无关文法生成合成语言，这些文法共享上层但不共享下层产生式规则。在此设置中，贝叶斯最优的下一个词预测器是信念传播（BP）；在连续层中编码其消息，产生分析预测，与在同一数据上训练的变换器（transformer）良好吻合。该框架解释了为何跨语言相似性在中间层达到峰值，与语言特定结构共存，并随语言接近度、模型质量和数据暴露而增强。它区分了相似性（共享的邻域几何）与对齐（共享的坐标），表明当代码切换数据（即混合语言句子）足够丰富时，后者会发生。它进一步预测，从每层中减去由前一层线性可预测的分量会增加跨语言相似性，我们在预训练的大语言模型（LLM）中证实了这一点。
英文摘要
Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
Comments10+14 pages, 7+12 figures
