发表机构
George Mason University（乔治梅森大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究发现仅解码器多语言模型中，双语预训练会使共享语言的深层隐藏状态表征存在差异，嵌入对齐无法掩盖该差异，这对依赖对齐模型的下游研究有重要影响。
AI 中文摘要
当研究人员比较用于探测、可解释性或跨语言迁移的多语言模型时，他们通常会对齐嵌入空间并假设共享语言的表征具有可比性。我们表明，对于仅解码器模型而言，这一假设可能为时过早。我们预训练了成对的3.1亿参数模型（一个仅英语，一个双语），覆盖8种类型学上多样的语言，分别控制英语暴露量、总计算量和文档重叠度。在对共享英语词汇对齐后，我们测试了预留词，发现 token 嵌入在对齐后看起来相似，但模型用于预测的更深层隐藏状态并非如此。这一差距在所有8种语言中都存在，且在控制文档重叠度和替代对齐方法后仍然存在。这种隐藏状态不匹配会在中间 Transformer 层中增大，表明它源于上下文处理而非执行对齐的输入表征。嵌入对齐可能会掩盖模型内部表征共享语言的真实差异，这对任何将对齐模型视为可互换的下游研究都很重要。
英文摘要
A concept can carry different associations across languages, while modern language models learn English alongside many other languages during pretraining. Yet comparisons among existing models cannot easily isolate how any one language changes the way these models represent English concepts because their training corpora, compute, architectures, and random seeds all differ. We study this question through a controlled experiment with 40 matched 310M-parameter decoder-only models that share an architecture, tokenizer, training recipe, and English data source. Each bilingual condition adds one of eight languages, while four experimental comparisons separately account for English exposure, total training, and English-document overlap. We align each model pair using 3,000 common English words, then measure where 1,000 held-out English concepts fall along 50 fixed semantic contrasts, such as red versus white. Across 32 experimental comparisons, English concept positions differ more between bilingual and English-only conditions than between English-only runs with different random seeds. These differences are larger in contextual states than in token embeddings and peak in middle layers. The language learned alongside English can therefore change how a model represents English concepts even when its English input representations are explicitly aligned.
CommentsPublished as a conference paper at EMNLP 2026 (Main)
