PALM：用于上下文条件生成的自编码自回归语言模型的预训练
PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
摘要：
自监督预训练，如BERT，MASS和BART，已经成为一种强大的自然语言理解和生成技术。现有的预训练技术采用自编码和/或自回归目标来训练基于transformer的模型，通过从具有一些掩蔽令牌的损坏文本中恢复原始单词令牌。现有技术的训练目标通常与许多语言生成任务的目标不一致，例如生成式问答和会话响应生成，用于产生给定上下文的新文本。这项工作提出了一种新的计划，共同预训练的自编码和自回归语言模型上的大型未标记的语料库，专门设计用于生成新的文本条件下的上下文PALM。新方案消除了现有去噪方案在预训练和微调之间引入的不匹配，其中生成多于重建原始文本。一组广泛的实验表明，PALM在各种语言生成基准测试中取得了新的最先进的结果，包括生成式问答（官方MARCO排行榜上的排名1），CNN/DailyMail以及Gigaword上的抽象摘要，SQuAD上的问题生成以及Cornell Movie Dialogues上的会话响应生成。
英文摘要：
Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
