面向多语言模型预训练的软语言聚类
Soft Language Clustering for Multilingual Model Pre-training
AI总结：
针对多语言预训练模型在语言类型差异大、预训练数据不足时性能受限的问题，提出XLM-P方法，通过上下文条件检索提示引导编码，实现轻量级跨语言知识建模与易集成性，在XTREME基准多任务上提升性能，尤其利好低资源与类型差异大的目标语言。
摘要：
多语言预训练语言模型已展现出出色的（零样本）跨语言迁移能力，但当目标语言与源语言类型差异较大，或预训练数据规模有限时，其性能会受到阻碍。本文提出XLM-P，该方法可根据上下文检索提示词，作为实例编码的灵活条件引导。XLM-P实现了两大特性：一是对跨语言的语言不变知识与语言特定知识进行轻量级建模；二是可轻松与其他多语言预训练方法集成。在XTREME基准的文本分类、序列标注、问答和句子检索任务上，采用本文方法预训练的基础版与大规模语言模型均取得了一致的性能提升。此外，该方法在无监督句子检索任务中对低资源语言，以及在跨语言迁移中对与源语言差异极大的目标语言，均具备显著优势。
英文摘要：
Multilingual pre-trained language models have demonstrated impressive (zero-shot) cross-lingual transfer abilities, however, their performance is hindered when the target language has distant typology from source languages or when pre-training data is limited in size. In this paper, we propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods. On the tasks of XTREME including text classification, sequence labeling, question answering, and sentence retrieval, both base- and large-size language models pre-trained with our proposed method exhibit consistent performance improvement. Furthermore, it provides substantial advantages for low-resource languages in unsupervised sentence retrieval and for target languages that differ greatly from the source language in cross-lingual transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
