arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1909.01953cs.CL
收藏

用于不同序列生成的混合内容选择

Mixture Content Selection for Diverse Sequence Generation

Allen Institute for AI(艾伦人工智能研究所);Clova AI;NAVER;University of Washington(华盛顿大学)

Jaemin Cho, Minjoon Seo, Hannaneh Hajishirzi

摘要:

生成不同的序列在许多NLP应用中非常重要,例如问题生成或摘要,这些应用在源序列和目标序列之间表现出语义上的一对多关系。我们提出了一种方法,明确分离多样化的生成使用一个通用的即插即用模块(称为选择器),包装和指导现有的编码器-解码器模型。多样化阶段使用专家的混合专家模型来对源序列上的不同二进制掩码进行采样以用于多样化内容选择。生成阶段使用标准的编码器-解码器模型,给定来自源序列的每个所选内容。由于离散采样的不可微性和二进制掩码缺乏地面真实标签,我们利用地面真实掩码的代理,并采用随机硬EM进行训练。在问题生成(SQuAD)和抽象摘要(CNN-DM)中,我们的方法在准确性,多样性和训练效率方面都有显着提高,包括两个数据集中最先进的top-1准确性,top-5准确性提高6%,并且训练速度比最先进的模型快3.7倍。我们的代码可在https://github.com/clovaai/FocusSeq2Seq上公开获取。

英文摘要:

Generating diverse sequences is important in many NLP applications such as question generation or summarization that exhibit semantically one-to-many relationships between source and the target sequences. We present a method to explicitly separate diversification from generation using a general plug-and-play module (called SELECTOR) that wraps around and guides an existing encoder-decoder model. The diversification stage uses a mixture of experts to sample different binary masks on the source sequence for diverse content selection. The generation stage uses a standard encoder-decoder model given each selected content from the source sequence. Due to the non-differentiable nature of discrete sampling and the lack of ground truth labels for binary mask, we leverage a proxy for ground truth mask and adopt stochastic hard-EM for training. In question generation (SQuAD) and abstractive summarization (CNN-DM), our method demonstrates significant improvements in accuracy, diversity and training efficiency, including state-of-the-art top-1 accuracy in both datasets, 6% gain in top-5 accuracy, and 3.7 times faster training over a state of the art model. Our code is publicly available at https://github.com/clovaai/FocusSeq2Seq.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑