将多语言理解模型外推为多语言生成器
Extrapolating Multilingual Understanding Models as Multilingual Generators
AI总结：
本文提出语义引导的对齐后去噪（SGA）方法，将多语言理解模型XLM-R适配为生成器，在机器翻译、问题生成和故事生成任务上显著优于初始化方法，但在监督设置下仍不及mBART。
摘要：
通过掩码语言建模预训练的多语言理解模型（或基于编码器的模型，如mBERT）在许多语言理解任务上取得了有希望的结果。然而，与自回归（AR）模型相比，这些非自回归（NAR）模型在生成高质量文本方面仍然存在困难。考虑到基于编码器的模型具有高效生成和自我纠正能力的优势，本文探索了赋予多语言理解模型生成能力以获得统一模型的方法。具体来说，我们从多语言编码器（XLM-R）出发，提出了一种语义引导的对齐后去噪（SGA）方法，以少量新参数将编码器适配为多语言生成器。实验表明，所提出的方法是一种有效的适配方法，在XLM-R_large上，机器翻译的BLEU提升了9.4，问题生成的Rouge-L提升了8.1，故事生成的METEOR提升了5.5，优于广泛使用的基于初始化的方法。另一方面，我们观察到尽管在零样本设置下结果更好，XLM-R在监督设置下仍不如mBART，表明需要更多探索以使理解模型成为强大的生成器。
英文摘要：
Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these non-autoregressive (NAR) models still struggle to generate high-quality texts compared with autoregressive (AR) models. Considering that encoder-based models have the advantage of efficient generation and self-correction abilities, this paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. Specifically, we start from a multilingual encoder (XLM-R) and propose a \textbf{S}emantic-\textbf{G}uided \textbf{A}lignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-R$_{large}$. On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
