探索多语言与双语翻译模型之间的表征差异
Exploring Representational Disparities Between Multilingual and Bilingual Translation Models
AI总结：
本文通过内在维度和IsoScore分析双语与多语言翻译模型的表征几何差异，发现多语言解码器表征各向同性更低且占用维度更少，其各向异性源于建模特定语言信息，进而限制了表征容量。
摘要：
多语言机器翻译通过完全的多语言参数共享，在许多语言对中展现出了极大的参数效率和整体性能优势。然而，多语言模型中的某些语言对性能可能不如双语模型，尤其是在一对多翻译设置中。受这些经验差异的启发，我们研究了双语模型与一对多多语言模型在表征上的几何差异。具体而言，我们使用内在维度和IsoScore计算这些表征的各向同性，以衡量表征如何利用其底层向量空间中的维度。在两个模型中使用相同的评估数据，我们发现对于给定的语言对，其多语言模型解码器表征的各向同性始终较低，并且占用的维度少于可比的双语模型解码器表征。此外，我们表明多语言解码器表征中的大部分各向异性可归因于建模特定语言信息，从而限制了剩余的表征容量。
英文摘要：
Multilingual machine translation has proven immensely useful for both parameter efficiency and overall performance across many language pairs via complete multilingual parameter sharing. However, some language pairs in multilingual models can see worse performance than in bilingual models, especially in the one-to-many translation setting. Motivated by their empirical differences, we examine the geometric differences in representations from bilingual models versus those from one-to-many multilingual models. Specifically, we compute the isotropy of these representations using intrinsic dimensionality and IsoScore, in order to measure how the representations utilize the dimensions in their underlying vector space. Using the same evaluation data in both models, we find that for a given language pair, its multilingual model decoder representations are consistently less isotropic and occupy fewer dimensions than comparable bilingual model decoder representations. Additionally, we show that much of the anisotropy in multilingual decoder representations can be attributed to modeling language-specific information, therefore limiting remaining representational capacity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
