数据约束下的多语言知识迁移：通过词汇干预
Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions
AI总结：
提出LINK方法，利用双语词汇对预训练数据中的高资源语言部分进行词汇替换，无需额外模型训练或大量平行数据，即可有效提升低资源语言的知识迁移效果。
摘要：
跨语言知识迁移对于为训练数据不足的语言构建高性能多语言语言模型至关重要。当目标语言数据稀缺时，许多涉及科学推理、常识推理和世界知识的下游任务所需的知识必须主要从高资源语言获取，因此有效的知识迁移至关重要。现有的改进此类跨语言知识迁移的方法需要大量平行数据、翻译系统、辅助模型或额外的训练阶段，而这些对于许多语言来说基本不可用。我们提出LINK——一种数据级干预方法，通过使用双语词汇对预训练数据中的高资源部分进行词汇替换，在模型预训练期间改进知识迁移。对于给定的替换比例，在高资源（英语）训练语料库的一部分中，随机选择的单词被替换为其词级翻译，无需额外的模型训练，仅需一个双语词汇，而该词汇几乎可以零成本获得。对五种模型规模下的八种语言的评估显示，目标语言的下游任务有显著改进，达到同等性能的训练速度最高可提升2倍。
英文摘要：
Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is scarce, the knowledge required for many downstream tasks involving scientific reasoning, commonsense inference, and world knowledge must be acquired primarily from the high-resource language, making effective knowledge transfer essential. Existing methods for improving such cross-lingual knowledge transfer require large amounts of parallel data, translation systems, auxiliary models, or additional training stages that are largely unavailable for many languages. We propose LINK - a data-level intervention method that improves knowledge transfer during model pretraining through lexical substitutions in high-resource part of pretraining data using bilingual vocabularies. For a given replacement ratio, randomly selected words in a portion of the high-resource (English) training corpus are swapped with their word-level translations, requiring no additional model training and only a bilingual vocabulary, which can be obtained at near-zero cost for virtually any language. Evaluation on eight languages across five model sizes shows notable improvements on downstream tasks in the target language, with up to a 2x speedup in training to reach equivalent performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
