arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.07535cs.CL
收藏

SPMoE:利用稀疏模式混合专家混合专家模型生成多模式感知输出

SPMoE: Generate Multiple Pattern-Aware Outputs with Sparse Pattern Mixture of Experts

DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)

Shaobo Cui, Xintong Bao, Xuming Lin, Zhongzhou Zhao, Ji Zhang, Wei Zhou, Haiqing Chen

摘要:

许多生成任务遵循一对多的映射关系:每个输入可以与多个输出相关联。现有的方法,如条件变分自编码器(CVAE),采用潜在变量来建模这种一对多关系。然而,这种高维度和密集的潜变量缺乏可解释性,通常会导致不良和不可控的世代。本文创新性地引入了模式的语言学概念,将一对多映射分解为多个一对一映射,并进一步提出了稀疏模式混合专家模型混合专家模型(SPMoE)。每个一对一映射都与条件生成模式相关联,并由SPMoE中的专家进行建模。为了保证每一种语言模式都能由一个专家模型来单独处理,以获得更好的解释性和多样性,SPMoE采用稀疏机制来协调所有的专家模型。我们评估了我们的SPMoE的性能上的释义生成任务和实验结果证明,SPMoE可以达到一个很好的平衡质量,模式级的多样性,语料库级的多样性。

英文摘要:

Many generation tasks follow a one-to-many mapping relationship: each input could be associated with multiple outputs. Existing methods like Conditional Variational AutoEncoder(CVAE) employ a latent variable to model this one-to-many relationship. However, this high-dimensional and dense latent variable lacks explainability and usually leads to poor and uncontrollable generations. In this paper, we innovatively introduce the linguistic concept of pattern to decompose the one-to-many mapping into multiple one-to-one mappings and further propose a model named Sparse Pattern Mixture of Experts(SPMoE). Each one-to-one mapping is associated with a conditional generation pattern and is modeled with an expert in SPMoE. To ensure each language pattern can be exclusively handled with an expert model for better explainability and diversity, a sparse mechanism is employed to coordinate all the expert models in SPMoE. We assess the performance of our SPMoE on the paraphrase generation task and the experiment results prove that SPMoE can achieve a good balance in terms of quality, pattern-level diversity, and corpus-level diversity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑