生成任务中有效零样本跨语言知识迁移的关键要素
Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks
AI总结：
本研究在统一设置下比较生成任务中零样本跨语言迁移的多种方法，发现学习率调优与全量微调是关键，简单基线即可媲美复杂方法并达到数据翻译上限。
摘要：
零样本跨语言知识迁移使多语言预训练语言模型在一种语言上微调某任务后，能为其他语言的该任务进行预测。尽管该设置在自然语言理解任务中被广泛研究，但在生成任务中研究不足。先前工作注意到生成错误语言的频繁问题并提出解决方法，通常以 mT5 为骨干模型。本研究在统一设置下比较文献中的各种方法，并纳入替代骨干模型 mBART 和 NLLB-200。我们首先强调微调学习率的重要性，它能大幅缓解生成错误语言的问题。然后表明，经仔细学习率调优后，简单的模型全量微调即可作为极强基线，替代方法仅带来边际提升。最后发现，mBART 表现与同等规模的 mT5 相当，NLLB-200 在部分场景具竞争力。最终零样本模型达到了基于数据翻译方法的性能，后者通常被视为生成任务中零样本跨语言迁移的上限基线。
英文摘要：
Zero-shot cross-lingual knowledge transfer enables a multilingual pretrained language model, finetuned on a task in one language, make predictions for this task in other languages. While being broadly studied for natural language understanding tasks, the described setting is understudied for generation. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final zero-shot models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual transfer in generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
