多语言Transformer的大规模多语言词汇特化
Massively Multilingual Lexical Specialization of Multilingual Transformers
AI总结：
本文利用BabelNet在50种语言上构建同义词对，通过对比目标对mBERT和XLM-R进行大规模多语言词汇特化，显著提升跨语言词汇与句子检索任务性能，并能泛化到特化中未见过的语言。
摘要：
虽然预训练语言模型（PLMs）主要作为通用文本编码器，可针对各种下游任务进行微调，但近期研究表明，它们也可以被重构以生成高质量的词语表示（即静态词嵌入），并在类型级词汇任务中取得良好性能。虽然现有工作主要关注使用大量单语约束对单语PLMs进行词汇特化，但在本工作中，我们利用BabelNet作为现成的多语言和跨语言类型级词汇知识丰富来源，将大规模多语言Transformer（MMTs，例如mBERT或XLM-R）暴露于大规模多语言词汇知识中。具体而言，我们使用BabelNet的多语言同义词集在50种语言中创建同义词对（或同义词-释义对），然后通过对比目标引导的词汇特化程序对MMTs（mBERT和XLM-R）进行特化。我们表明，这种大规模多语言词汇特化在两个标准跨语言词汇任务（双语词典归纳和跨语言词语相似度）以及跨语言句子检索中都带来了显著提升。至关重要的是，我们观察到在特化过程中未见过的语言上也获得了提升，这表明多语言词汇特化能够泛化到没有词汇约束的语言。在一系列后续对照实验中，我们表明特化约束的数量比其来源语言集合所起的作用要大得多。
英文摘要：
While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-quality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks. While existing work primarily focused on the lexical specialization of monolingual PLMs with immense quantities of monolingual constraints, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multilingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lexical knowledge. Concretely, we use BabelNet's multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 languages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization procedure guided by a contrastive objective. We show that such massively multilingual lexical specialization brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon induction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval. Crucially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints. In a series of subsequent controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
