发现用于多语言预训练的表示Sprachbund
Discovering Representation Sprachbund For Multilingual Pre-Training
摘要：
多语言预训练模型已经在许多多语言NLP任务中证明了它们的有效性，并实现了从高资源语言到低资源语言的zero-shot或Few-Shot迁移。然而，由于某些语言之间存在显著的类型差异和矛盾，这些模型通常在许多语言和跨语言环境中表现不佳，这表明学习单一模型同时处理大量不同语言的难度。为了缓解这个问题，我们提出了一个新的多语言预训练管道。我们建议从多语言预训练模型中生成语言表示，并进行语言分析，以表明语言表示相似性从多个角度反映语言相似性，包括语言家族，地理sprachbund，词汇统计和句法。然后将所有的目标语言聚类为多个组，并将每个组命名为一个表示语言。因此，在相同的表示sprachbund的语言应该在预训练和微调中相互促进，因为它们具有丰富的语言相似性。我们为每个表示sprachbund预训练一个多语言模型。在跨语言基准上进行了实验，与强基线相比，取得了显着的改进。
英文摘要：
Multilingual pre-trained models have demonstrated their effectiveness in many multilingual NLP tasks and enabled zero-shot or few-shot transfer from high-resource languages to low resource ones. However, due to significant typological differences and contradictions between some languages, such models usually perform poorly on many languages and cross-lingual settings, which shows the difficulty of learning a single model to handle massive diverse languages well at the same time. To alleviate this issue, we present a new multilingual pre-training pipeline. We propose to generate language representation from multilingual pre-trained models and conduct linguistic analysis to show that language representation similarity reflect linguistic similarity from multiple perspectives, including language family, geographical sprachbund, lexicostatistics and syntax. Then we cluster all the target languages into multiple groups and name each group as a representation sprachbund. Thus, languages in the same representation sprachbund are supposed to boost each other in both pre-training and fine-tuning as they share rich linguistic similarity. We pre-train one multilingual model for each representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
