arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.16336cs.CLcs.LG
收藏

学习稀疏原型生成文本

Learning Sparse Prototypes for Text Generation

Carnegie Mellon University(卡内基梅隆大学);University of California San Diego(加州大学圣地亚哥分校)

Junxian He, Taylor Berg-Kirkpatrick, Graham Neubig

摘要:

原型驱动的文本生成使用非参数模型,首先从句子“原型”库中选择,然后修改原型以生成输出文本。虽然有效,但这些方法在测试时效率低下,因为需要存储和索引整个训练语料库。此外,现有的方法通常需要机器学习来识别在训练时参考哪些原型。在本文中,我们提出了一种新的生成模型,自动学习稀疏的原型支持集,尽管如此,实现强大的语言建模性能。这是通过(1)在原型选择分布上施加稀疏诱导先验,以及(2)利用摊销变分推理来学习原型检索函数来实现的。在实验中,我们的模型优于以前的原型驱动的语言模型,同时实现了高达1000倍的内存减少,以及1000倍的速度在测试时。更有趣的是,我们表明,学习的原型能够捕捉语义和语法在不同的粒度,因为我们不同的稀疏性的原型选择,某些句子属性可以通过指定的原型生成控制。

英文摘要:

Prototype-driven text generation uses non-parametric models that first choose from a library of sentence "prototypes" and then modify the prototype to generate the output text. While effective, these methods are inefficient at test time as a result of needing to store and index the entire training corpus. Further, existing methods often require heuristics to identify which prototypes to reference at training time. In this paper, we propose a novel generative model that automatically learns a sparse prototype support set that, nonetheless, achieves strong language modeling performance. This is achieved by (1) imposing a sparsity-inducing prior on the prototype selection distribution, and (2) utilizing amortized variational inference to learn a prototype retrieval function. In experiments, our model outperforms previous prototype-driven language models while achieving up to a 1000x memory reduction, as well as a 1000x speed-up at test time. More interestingly, we show that the learned prototypes are able to capture semantics and syntax at different granularity as we vary the sparsity of prototype selection, and that certain sentence attributes can be controlled by specifying the prototype for generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑