Generating diverse sequences is important in many NLP applications such as question generation or summarization that exhibit semantically one-to-many relationships between source and the target sequences. We present a method to explicitly separate diversification from generation using a general plug-and-play module (called SELECTOR) that wraps around and guides an existing encoder-decoder model. The diversification stage uses a mixture of experts to sample different binary masks on the source sequence for diverse content selection. The generation stage uses a standard encoder-decoder model given each selected content from the source sequence. Due to the non-differentiable nature of discrete sampling and the lack of ground truth labels for binary mask, we leverage a proxy for ground truth mask and adopt stochastic hard-EM for training. In question generation (SQuAD) and abstractive summarization (CNN-DM), our method demonstrates significant improvements in accuracy, diversity and training efficiency, including state-of-the-art top-1 accuracy in both datasets, 6% gain in top-5 accuracy, and 3.7 times faster training over a state of the art model. Our code is publicly available at https://github.com/clovaai/FocusSeq2Seq.
Mixture Models for Diverse Machine Translation: Tricks of the Trade
MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Facebook AI Research(脸书人工智能研究院)
通过EM训练的混合模型是机器学习文献中最简单、最广泛使用和最容易理解的潜在变量模型之一。令人惊讶的是,这些模型几乎没有在文本生成应用程序,如机器翻译。原则上,它们提供了一个潜在的变量来控制生成,并产生一组不同的假设。然而,在实践中,混合模型很容易退化--通常只有一个组件得到训练,或者潜在变量被简单地忽略。我们发现,在责任计算中禁用丢弃噪声对成功训练至关重要。此外,参数化,先验分布,硬与软EM和在线与离线分配的设计选择可以显着影响模型的性能。我们开发了一个评估协议,以评估质量和多样性的代对多个参考,并提供了一个广泛的实证研究的几个混合模型的变体。我们的分析表明,某些类型的混合模型是更强大的,并提供最好的权衡翻译质量和多样性相比,变分模型和不同的解码方法。\footnote{Code to reproduce the results in this paper is available at \url{https://github.com/pytorch/fairseq}}
Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts
Amirkabir University of Technology (Tehran Polytechnic)(德黑兰理工大学);Department of Mathematics and Computer Science(数学与计算机科学系);Department of Computer Engineering(计算机工程系)
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)