通过基于转写的训练后对齐打破多语言预训练语言模型中的文字壁垒
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment
AI总结：
针对不同文字的语言间跨语言迁移性能受限的问题，提出基于转写的预训练后对齐方法，在两个多文字区域语言组上验证，可显著提升跨语言迁移效果，非英语源语言场景下提升更明显。
摘要：
多语言预训练模型（mPLMs）在跨语言迁移任务上已展现出优异性能。然而，当低资源目标语言与高资源源语言使用不同文字书写时，即便两种语言可能存在关联或共享部分词汇，迁移性能往往会受到阻碍。受近期利用转写解决该问题的研究启发，本文提出一种基于转写的预训练后对齐（PPA）方法，旨在提升使用不同文字的语言之间的跨语言对齐效果。我们选取了两个区域语言组：地中海-阿姆哈拉语-波斯语（Mediterranean-Amharic-Farsi）组与东南亚及东亚语言（South+East Asian Languages）组，组内语言相互影响但使用不同文字。我们将所提方法应用于这些语言组，并在一系列下游任务上开展了广泛实验。结果表明，经过PPA后，模型在以英语为中心的迁移中性能始终优于原模型，部分任务提升幅度最高达50%。此外，当我们使用英语以外的语言作为迁移源语言时，该方法取得的提升幅度更大。我们将在https://github.com/cisnlp/Transliteration-PPA 公开发布代码与模型。
英文摘要：
Multilingual pre-trained models (mPLMs) have shown impressive performance on cross-lingual transfer tasks. However, the transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language, even though the two languages may be related or share parts of their vocabularies. Inspired by recent work that uses transliteration to address this problem, our paper proposes a transliteration-based post-pretraining alignment (PPA) method aiming to improve the cross-lingual alignment between languages using diverse scripts. We select two areal language groups, $\textbf{Mediterranean-Amharic-Farsi}$ and $\textbf{South+East Asian Languages}$, wherein the languages are mutually influenced but use different scripts. We apply our method to these language groups and conduct extensive experiments on a spectrum of downstream tasks. The results show that after PPA, models consistently outperform the original model (up to 50% for some tasks) in English-centric transfer. In addition, when we use languages other than English as sources in transfer, our method obtains even larger improvements. We will make our code and models publicly available at \url{https://github.com/cisnlp/Transliteration-PPA}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
