arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.08067cs.CL
收藏

面向任务对话系统的层次化Transformer

Hierarchical Transformer for Task Oriented Dialog Systems

Indian Institute of Technology Kharagpur(印度理工学院卡哈拉格普尔分校)

Bishal Santra, Potnuru Anusha, Pawan Goyal

摘要:

由于最近基于RNN和Transformer的模型在问答和摘要等任务中的成功,对话系统的生成模型已经引起了人们的极大兴趣。虽然对话响应生成的任务通常被视为序列到序列(Seq 2Seq)问题,但是过去的研究人员发现使用标准Seq 2Seq模型来训练对话系统具有挑战性。因此,为了帮助模型学习有意义的话语和会话级别的特征,Sordoni et al.(2015 b); Serban et al.(2016)提出了分层RNN架构,后来被其他几个基于RNN的对话系统采用。随着基于transformer的模型最近主导seq 2seq问题,自然要问的问题是层次概念在基于Transformer的对话系统中的适用性。在本文中,我们提出了一个通用的框架分层Transformer编码器,并显示如何一个标准的Transformer可以变形为任何分层编码器,包括HRED和HIBERT的模型,通过使用专门设计的注意掩模和位置编码。我们证明了层次编码有助于实现更好的自然语言的上下文理解的transformer为基础的模型,面向任务的对话系统,通过广泛的实验。

英文摘要:

Generative models for dialog systems have gained much interest because of the recent success of RNN and Transformer based models in tasks like question answering and summarization. Although the task of dialog response generation is generally seen as a sequence-to-sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn meaningful utterance and conversation level features, Sordoni et al. (2015b); Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer-based models dominating the seq2seq problems lately, the natural question to ask is the applicability of the notion of hierarchy in transformer based dialog systems. In this paper, we propose a generalized framework for Hierarchical Transformer Encoders and show how a standard transformer can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog systems through a wide range of experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑