上下文词表示跨语言调整对零样本迁移的影响
The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
AI总结：
本文研究跨语言调整上下文词表示对零样本迁移的影响，在西班牙语、俄语、越南语和印地语上扩展了 mBERT 跨语言调整方法至 XSR、NER 和 QA 任务，发现微调会遗忘部分跨语言对齐信息，并通过持续学习进一步提升了 NLI 性能。
摘要：
大型多语言语言模型（如 mBERT 或 XLM-R）能够在各种信息检索和自然语言处理任务中实现零样本跨语言迁移。Cao 等人（2020）提出了一种数据和计算高效的 mBERT 跨语言调整方法，该方法使用小型平行语料库使不同语言中相关词的嵌入彼此相似。他们展示了该方法在五种欧洲语言的自然语言推理（NLI）任务中的有效性。与之相比，我们在类型学上更多样的语言集合（西班牙语、俄语、越南语和印地语）上进行实验，并将其原始实现扩展到新任务（跨语言句子检索 XSR、命名实体识别 NER 和问答 QA）以及额外的训练机制（持续学习）。我们的研究在四种语言上复现了 NLI 任务的提升，在三种语言上展示了 NER、XSR 和跨语言 QA 结果的改善（尽管部分跨语言 QA 提升在统计上不显著），而单语 QA 性能从未提升，有时甚至下降。对相关词和不相关词（跨语言）的上下文嵌入之间距离的分析表明，微调会导致部分跨语言对齐信息被“遗忘”。基于这一观察，我们利用持续学习进一步提升了 NLI 性能。
英文摘要：
Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. They showed it to be effective in NLI for five European languages. In contrast we experiment with a typologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks (XSR, NER, and QA) and an additional training regime (continual learning). Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA results in three languages (though some cross-lingual QA gains were not statistically significant), while mono-lingual QA performance never improved and sometimes degraded. Analysis of distances between contextualized embeddings of related and unrelated words (across languages) showed that fine-tuning leads to "forgetting" some of the cross-lingual alignment information. Based on this observation, we further improved NLI performance using continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
