arXiv 2310.099172024-04-22cs.CL

预训练多语言语言模型在生成任务中零样本跨语言知识转移的实证研究

Empirical study of pretrained multilingual language models for zero-shot cross-lingual knowledge transfer in generation

Naver Labs Europe（Naver欧洲实验室）；LMU Munich（慕尼黑大学）

Nadezhda Chirkova, Sheng Liang, Vassilina Nikoulina

备注：: This preprint describes a preliminary study for our follow-up work arXiv:2402.12279 (NAACL 2024), in which we investigate important factors for enabling zero-shot cross-lingual transfer in generative tasks

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文实证研究了mBART和NLLB-200等替代多语言预训练模型在生成任务零样本跨语言知识转移中的表现，发现带适配器的mBART与同规模mT5相当，且调整微调学习率能有效缓解生成错误语言的问题。

摘要：

零样本跨语言知识转移使得在一种语言上针对某任务进行微调的多语言预训练语言模型（mPLM）能够对该任务在其他语言下进行预测。尽管该设定在自然语言理解任务中被广泛研究，但在生成任务中仍研究不足。以往工作注意到生成错误语言的频繁问题，并通常使用mT5作为骨干模型提出解决方法。本研究测试了mBART和NLLB-200等替代mPLM，考虑了全量微调及使用适配器的参数高效微调。结果表明，带适配器的mBART与同规模mT5表现相当，NLLB-200在某些情况下具竞争力。此外，我们强调调整微调学习率的重要性，这有助于缓解生成错误语言的问题。

英文摘要：

Zero-shot cross-lingual knowledge transfer enables the multilingual pretrained language model (mPLM), finetuned on a task in one language, make predictions for this task in other languages. While being broadly studied for natural language understanding tasks, the described setting is understudied for generation. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work, we test alternative mPLMs, such as mBART and NLLB-200, considering full finetuning and parameter-efficient finetuning with adapters. We find that mBART with adapters performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. We also underline the importance of tuning learning rate used for finetuning, which helps to alleviate the problem of generation in the wrong language.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。