arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1911.03014cs.CLcs.LG
收藏

用于神经文本生成的复活子模块

Resurrecting Submodularity for Neural Text Generation

Nanyang Technological University(南洋理工大学);Salesforce Research Asia(Salesforce亚洲研究院)

Simeng Han, Xiang Lin, Shafiq Joty

摘要:

子模块化对于内容选择中的各种目标是期望的,其中当前的神经编码器-解码器框架是不足的。然而,到目前为止,它还没有被探索在神经编码器-解码器系统的文本生成。在这项工作中,我们定义了减少注意力与子模块功能,反过来,证明了有效的神经覆盖的子模块性。贪心算法近似求解子模最大化问题,不适用于自回归生成中的注意力分数优化。因此,而不是遵循如何子模函数已被广泛使用,我们提出了一个简化的,但原则性的解决方案。由此产生的注意力模块提供了一种架构简单且经验有效的方法来提高神经文本生成的覆盖率。我们在三个有向文本生成任务上进行了实验,这些任务具有不同的恢复率水平,跨越两种模式,三种不同的神经模型架构和两种训练策略变体。结果和分析表明,我们的方法在这些设置中推广良好,产生高质量的文本,并优于最先进的基线。

英文摘要:

Submodularity is desirable for a variety of objectives in content selection where the current neural encoder-decoder framework is inadequate. However, it has so far not been explored in the neural encoder-decoder system for text generation. In this work, we define diminishing attentions with submodular functions and in turn, prove the submodularity of the effective neural coverage. The greedy algorithm approximating the solution to the submodular maximization problem is not suited to attention score optimization in auto-regressive generation. Therefore instead of following how submodular function has been widely used, we propose a simplified yet principled solution. The resulting attention module offers an architecturally simple and empirically effective method to improve the coverage of neural text generation. We run experiments on three directed text generation tasks with different levels of recovering rate, across two modalities, three different neural model architectures and two training strategy variations. The results and analyses demonstrate that our method generalizes well across these settings, produces texts of good quality and outperforms state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑