探索多语言 Transformer 中对齐与跨语言迁移之间的关系
Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
AI总结：
本文研究多语言 Transformer 中表示对齐与跨语言迁移的关系，发现二者显著相关，并明确重对齐在小模型、远距离语言和词典词对等条件下可显著提升性能。
摘要：
在没有任何显式跨语言训练数据的情况下，多语言语言模型也能够实现跨语言迁移。提高这种迁移的一种常见方法是在微调之前执行重对齐步骤，即训练模型为来自翻译句子的词对构建相似的表示。但研究发现，这类重对齐方法并非总能提升不同语言和任务上的结果，这引出了一个问题：对齐表示是否真的有益于跨语言迁移。我们提供证据表明，在不同语言、模型和随机种子之间，对齐实际上与跨语言迁移显著相关。我们表明，微调会对对齐产生显著影响，这主要取决于下游任务和模型。最后，我们表明重对齐在某些情况下可以改善跨语言迁移，并确定了重对齐方法带来显著提升的条件。具体而言，我们发现，当泛化到远距离语言、使用较小模型，以及使用双语词典而非 FastAlign 来提取重对齐词对时，在对齐与跨语言迁移相关的任务上，重对齐效果更好。例如，对于英语和阿拉伯语之间的 POS-tagging（词性标注），重对齐可以在 distilmBERT 上带来 +15.8 的准确率提升，甚至比 XLM-R Large 高出 1.7。因此，我们主张进一步研究面向较小多语言模型的重对齐方法，将其作为扩展规模的替代方案。
英文摘要：
Without any explicit cross-lingual training data, multilingual language models can achieve cross-lingual transfer. One common way to improve this transfer is to perform realignment steps before fine-tuning, i.e., to train the model to build similar representations for pairs of words from translated sentences. But such realignment methods were found to not always improve results across languages and tasks, which raises the question of whether aligned representations are truly beneficial for cross-lingual transfer. We provide evidence that alignment is actually significantly correlated with cross-lingual transfer across languages, models and random seeds. We show that fine-tuning can have a significant impact on alignment, depending mainly on the downstream task and the model. Finally, we show that realignment can, in some instances, improve cross-lingual transfer, and we identify conditions in which realignment methods provide significant improvements. Namely, we find that realignment works better on tasks for which alignment is correlated with cross-lingual transfer when generalizing to a distant language and with smaller models, as well as when using a bilingual dictionary rather than FastAlign to extract realignment pairs. For example, for POS-tagging, between English and Arabic, realignment can bring a +15.8 accuracy improvement on distilmBERT, even outperforming XLM-R Large by 1.7. We thus advocate for further research on realignment methods for smaller multilingual models as an alternative to scaling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
