大规模多语言神经机器翻译中的注意力 sinks：发现、分析与缓解
Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation
AI总结：
研究发现神经机器翻译中跨注意力模式存在注意力 sinks，非内容token占据大部分注意力质量，影响相似性评估，提出内容过滤方法以恢复语言信号。
摘要：
神经机器翻译（NMT）中的交叉注意力模式被广泛用于研究多语言模型如何对齐语言结构。我们报告了在NLLB-200（6亿词）的交叉注意力分析中存在系统性偏差：非内容token（主要为结束符、语言标签和标点）捕获了总交叉注意力质量的83%至91%。我们称这些为“注意力 sinks”，将LLM中的发现扩展到NMT交叉注意力，并识别出词汇设计而非位置偏见是根本原因。此偏差导致原始指标低估内容层面相似性近一半（36.7% vs 70.7%），使未经修正的分析不可靠。为解决此问题，我们验证了一种仅内容过滤方法，去除非内容token并重新归一化分布。将其应用于1000个平行句子（非洲语言：斯瓦希里语、基库尤语、索马里语、卢语；非非洲基准：德语、土耳其语、中文、印地语），确认此偏差是普遍存在的，并恢复了被掩蔽的语言信号：教师强制与生成模式之间有16.9个百分点的差距，注意力熵中清晰的语言家族聚类，以及隐藏的索马里悖论，将SOV词序与单调对齐联系起来。我们发布了我们的过滤工具包和修正数据集，以支持多语言NMT的可重复解释性研究。
英文摘要：
Cross-attention patterns in neural machine translation (NMT) are widely used to study how multilingual models align linguistic structure. We report a systematic artifact in cross-attention analysis of NLLB-200 (600M): non-content tokens - primarily end-of-sequence tokens, language tags, and punctuation - capture 83 percent to 91 percent of total cross-attention mass. We term these "attention sinks," extending findings from LLMs [Xiao et al., 2023] to NMT cross-attention and identifying a causal mechanism rooted in vocabulary design rather than position bias. This artifact causes raw metrics to underestimate content-level similarity by nearly half (36.7 percent raw vs. 70.7 percent filtered), rendering uncorrected analyses unreliable. To address this, we validate a content-only filtering methodology that removes non-content tokens and renormalizes the distribution. Applying this to 1,000 parallel sentences across African languages (Swahili, Kikuyu, Somali, Luo) and non-African benchmarks (German, Turkish, Chinese, Hindi), we confirm the artifact is universal and recover masked linguistic signals: a 16.9 percentage-point gap between teacher-forcing and generation modes, clear language-family clustering in attention entropy, and a hidden Somali paradox linking SOV word order to monotonic alignment. We release our filtering toolkit and corrected datasets to support reproducible interpretability research on multilingual NMT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
