Sample-and-rank is a key decoding strategy for modern generation-based dialogue systems. It helps achieve diverse and high-quality responses by selecting an answer from a small pool of generated candidates. The current state-of-the-art ranking methods mainly use an encoding paradigm called Cross-Encoder, which separately encodes each context-candidate pair and ranks the candidates according to their fitness scores. However, Cross-Encoder repeatedly encodes the same lengthy context for each candidate, resulting in high computational costs. Poly-Encoder addresses the above problems by reducing the interaction between context and candidates, but with a price of performance drop. In this work, we develop a new paradigm called Uni-Encoder, that keeps the full attention over each pair as in Cross-Encoder while only encoding the context once, as in Poly-Encoder. Uni-Encoder encodes all the candidates with the context in one forward pass. We use the same positional embedding for all candidates to ensure they are treated equally and design a new attention mechanism to avoid confusion. Our Uni-Encoder can simulate other ranking paradigms using different attention and response concatenation methods. Extensive experiments show that our proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency. For instance, it improves R10@1 by 2.9% with an approximately 4X faster inference speed on the Ubuntu V2 dataset.
Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Beijing Institute of Technology（北京理工大学）；Huawei Technologies Ltd.（华为技术有限公司）
Complex dialogue mappings (CDM), including one-to-many and many-to-one mappings, tend to make dialogue models generate incoherent or dull responses, and modeling these mappings remains a huge challenge for neural dialogue systems. To alleviate these problems, methods like introducing external information, reconstructing the optimization function, and manipulating data samples are proposed, while they primarily focus on avoiding training with CDM, inevitably weakening the model's ability of understanding CDM in human conversations and limiting further improvements in model performance. This pap…
Recently, researchers have explored using the encoder-decoder framework to tackle dialogue state tracking (DST), which is a key component of task-oriented dialogue systems. However, they regard a multi-turn dialogue as a flat sequence, failing to focus on useful information when the sequence is long. In this paper, we propose a Hierarchical Dynamic Copy Network (HDCN) to facilitate focusing on the most informative turn, making it easier to extract slot values from the dialogue context. Based on the encoder-decoder framework, we adopt a hierarchical copy approach that calculates two levels of a…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tencent Cloud Xiaowei（腾讯云小微）
Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, w…
Modern day conversational agents are trained to emulate the manner in which humans communicate. To emotionally bond with the user, these virtual agents need to be aware of the affective state of the user. Transformers are the recent state of the art in sequence-to-sequence learning that involves training an encoder-decoder model with word embeddings from utterance-response pairs. We propose an emotion-aware transformer encoder for capturing the emotional quotient in the user utterance in order to generate human-like empathetic responses. The contributions of our paper are as follows: 1) An emo…
Copy-Enhanced Heterogeneous Information Learning for Dialogue State Tracking
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Unisound Information Technology Co., Ltd（北京云知声信息技术有限公司）
对话状态跟踪（DST）是面向任务的对话系统中的一个重要组成部分，它估计用户在每个对话回合的目标。然而，大多数先前的方法通常遭受以下问题。许多判别模型，特别是端到端（E2 E）模型，很难提取出候选本体中不存在的未知值;而以往的生成模型，虽然可以从话语中提取未知值，但由于忽略了预定义本体的语义信息，导致性能下降。此外，以前的生成模型通常需要手工制作的列表来规范化生成的值。如何整合预定义本体和对话文本（异构文本）的语义信息以生成未知值并提高性能成为一个严峻的挑战。本文提出了一种基于多编码器-解码器的复制增强异构信息学习模型CEDST（Copy Enhanced Heterogeneous Information Learning Model with Multiple Encoder-Decoder for DST），该模型通过从异构文本中复制值，有效地生成包括未知值在内的所有可能值。同时，CEDST通过多个编码器有效地将大的状态空间分解为若干个小的状态空间，并采用多个解码器充分利用这些小的状态空间来产生值。多编解码器架构可以显著提高性能。实验表明，CEDST可以达到最先进的结果在两个数据集和我们构造的数据集与许多未知值。
Task-oriented dialogue systems help users accomplish tasks such as booking a movie ticket and ordering food via conversation. Generative models parameterized by a deep neural network are widely used for next turn response generation in such systems. It is natural for users of the system to want to accomplish multiple tasks within the same conversation, but the ability of generative models to compose multiple tasks is not well studied. In this work, we begin by studying the effect of training human-human task-oriented dialogues towards improving the ability to compose multiple tasks on Transfor…
Dynamic Knowledge Fusion for Multi-Domain Dialogue State Tracking
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd（中国电信人工智能研究院（TeleAI））
University of Amsterdam（阿姆斯特丹大学）；Microsoft（微软）；Ahold Delhaize（阿霍德德尔海兹）
Being able to generate informative and coherent dialogue responses is crucial when designing human-like open-domain dialogue systems. Encoder-decoder-based dialogue models tend to produce generic and dull responses during the decoding step because the most predictable response is likely to be a non-informative response instead of the most suitable one. To alleviate this problem, we propose to train the generation model in a bidirectional manner by adding a backward reasoning step to the vanilla encoder-decoder training. The proposed backward reasoning step pushes the model to produce more info…
Microsoft Corporation（微软公司）；Peking University（北京大学）
Establishing retrieval-based dialogue systems that can select appropriate responses from the pre-built index has gained increasing attention from researchers. For this task, the adoption of pre-trained language models (such as BERT) has led to remarkable progress in a number of benchmarks. There exist two common approaches, including cross-encoders which perform full attention over the inputs, and bi-encoders that encode the context and response separately. The former gives considerable improvements in accuracy but is often inapplicable in practice for large-scale retrieval given the cost of t…
National University of Singapore（新加坡国立大学）；Universidad Politécnica de Madrid（马德里理工大学）；Robert Bosch (SEA) Pte Ltd（博世（东南亚）私人有限公司）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat, exchange of information, and grounding in emotional situations. To effectively measure the quality of such conversational agents, a model-based automatic dialogue evaluation metric (ADEM) is expected to perform well across multiple domains. Despite significant progress, an ADEM that works well in one domain does not necessarily generalize to another. This calls for a dedicated network architecture for domain generalization. To tackle the multi-domain dialogue evaluation task, we propose a Panel of Expe…
NTT Communication Science Laboratories（日本电信电话公司通信科学实验室）
In recent years, several high-performance conversational systems have been proposed based on the Transformer encoder-decoder model. Although previous studies analyzed the effects of the model parameters and the decoding method on subjective dialogue evaluations with overall metrics, they did not analyze how the differences of fine-tuning datasets affect on user's detailed impression. In addition, the Transformer-based approach has only been verified for English, not for such languages with large inter-language distances as Japanese. In this study, we develop large-scale Transformer-based Japan…
Sun Yat-sen University（中山大学）；Microsoft Corporation（微软公司）
Though recent end-to-end neural models have shown promising progress on Conversational Recommender System (CRS), two key challenges still remain. First, the recommended items cannot be always incorporated into the generated replies precisely and appropriately. Second, only the items mentioned in the training corpus have a chance to be recommended in the conversation. To tackle these challenges, we introduce a novel framework called NTRD for recommender dialogue system that decouples the dialogue generation from the item recommendation. NTRD has two key components, i.e., response template gener…
University College London（伦敦大学学院）；ByteDance AI Lab（字节跳动人工智能实验室）
This paper is concerned with dialogue state tracking (DST) in a task-oriented dialogue system. Building a DST module that is highly effective is still a challenging issue, although significant progresses have been made recently. This paper proposes a new approach to dialogue state tracking, referred to as Seq2Seq-DU, which formalizes DST as a sequence-to-sequence problem. Seq2Seq-DU employs two BERT-based encoders to respectively encode the utterances in the dialogue and the descriptions of schemas, an attender to calculate attentions between the utterance embeddings and the schema embeddings,…
The Hong Kong University of Science and Technology（香港科技大学）；Facebook；Columbia University（哥伦比亚大学）
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle task-oriented dialogue in unseen domains without the expense of collecting in-domain data. In this paper, we propose a slot description enhanced generative approach for zero-shot cross-domain DST. Specifically, our model first encodes dialogue context and slots with a pre-trained self-attentive encoder, and generates slot values in an auto-regressive manner. In addition, we incorporate Slot Type Informed Descriptions that capture the shared information across slots to facilitate cross-domain knowledge transfer. Experimen…
POSTECH（浦项科技大学）；Graduate School of Artificial Intelligence（人工智能研究生院）
Dialogue state tracking (DST) is an essential sub-task for task-oriented dialogue systems. Recent work has focused on deep neural models for DST. However, the neural models require a large dataset for training. Furthermore, applying them to another domain needs a new dataset because the neural models are generally trained to imitate the given dataset. In this paper, we propose Schema Encoding for Transferable Dialogue State Tracking (SETDST), which is a neural DST method for effective transfer to new domains. Transferable DST could assist developments of dialogue systems even with few dataset …
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）
Knowledge-grounded dialogue systems are challenging to build due to the lack of training data and heterogeneous knowledge sources. Existing systems perform poorly on unseen topics due to limited topics covered in the training data. In addition, heterogeneous knowledge sources make it challenging for systems to generalize to other tasks because knowledge sources in different knowledge representations require different knowledge encoders. To address these challenges, we present PLUG, a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge…
Explicit State Tracking with Semi-Supervision for Neural Dialogue Generation
Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
对话生成的任务旨在自动提供响应给定先前的话语。跟踪对话状态是判断用户意图的一个重要环节。然而，\emph{expensive nature of state labeling}和\emph{weak interpretability}使得对话状态跟踪对于面向任务和非面向任务的对话生成都是具有挑战性的问题：对于在面向任务的对话中生成响应，状态跟踪通常从人工注释的语料库中学习，其中人工注释对于训练来说是昂贵的;对于在非面向任务的对话中生成响应，由于对话状态的数量不受限制，大多数现有工作忽略了显式状态跟踪。在本文中，我们提出了\emph{semi-supervised explicit dialogue state tracker}（SEDST）的神经对话生成。为此，我们的方法有两个核心成分：\emph{CopyFlowNet}和\emph{posterior regularization}。具体来说，我们提出了一个编码器-解码器架构，命名为\emph{CopyFlowNet}，表示一个明确的对话状态的概率分布在词汇空间。为了优化训练过程，我们采用后验正则化策略来整合间接监督。在任务型和非任务型对话语料库上进行的大量实验证明了该模型的有效性。此外，我们发现，我们提出的半监督对话状态跟踪器实现了国家的最先进的监督学习基线状态跟踪过程中的性能相当。
University of Science and Technology of China（中国科学技术大学）；Queen’s University（女王大学）
Task-oriented conversational modeling with unstructured knowledge access, as track 1 of the 9th Dialogue System Technology Challenges (DSTC 9), requests to build a system to generate response given dialogue history and knowledge access. This challenge can be separated into three subtasks, (1) knowledge-seeking turn detection, (2) knowledge selection, and (3) knowledge-grounded response generation. We use pre-trained language models, ELECTRA and RoBERTa, as our base encoder for different subtasks. For subtask 1 and 2, the coarse-grained information like domain and entity are used to enhance kno…
With the rapid development of artificial intelligence, conversational bots have became prevalent in mainstream E-commerce platforms, which can provide convenient customer service timely. To satisfy the user, the conversational bots need to understand the user's intention, detect the user's emotion, and extract the key entities from the conversational utterances. However, understanding dialogues is regarded as a very challenging task. Different from common language understanding, utterances in dialogues appear alternately from different roles and are usually organized as hierarchical structures…
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Peng Cheng Laboratory（鹏城实验室）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海市智能信息处理重点实验室）
In dialogue systems, utterances with similar semantics may have distinctive emotions under different contexts. Therefore, modeling long-range contextual emotional relationships with speaker dependency plays a crucial part in dialogue emotion recognition. Meanwhile, distinguishing the different emotion categories is non-trivial since they usually have semantically similar sentiments. To this end, we adopt supervised contrastive learning to make different emotions mutually exclusive to identify similar emotions better. Meanwhile, we utilize an auxiliary response generation task to enhance the mo…
University of Science and Technology of China（中国科学技术大学）
With the advances in deep learning, tremendous progress has been made with chit-chat dialogue systems and task-oriented dialogue systems. However, these two systems are often tackled separately in current methods. To achieve more natural interaction with humans, a dialogue agent needs to be capable of both chatting and accomplishing tasks. To this end, we propose a unified dialogue system (UniDS) with the two aforementioned skills. In particular, we design a unified dialogue data schema, compatible for both chit-chat and task-oriented dialogues, and we train UniDS with mixed dialogue data from…
University of Oxford（牛津大学）；Tencent（腾讯）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；ByteDance（字节跳动）；Amazon Web Services（亚马逊云科技）
Most existing neural network based task-oriented dialogue systems follow encoder-decoder paradigm, where the decoder purely depends on the source texts to generate a sequence of words, usually suffering from instability and poor readability. Inspired by the traditional template-based generation approaches, we propose a template-guided hybrid pointer network for the knowledge-based task-oriented dialogue system, which retrieves several potentially relevant answers from a pre-constructed domain-specific conversational repository as guidance answers, and incorporates the guidance answers into bot…
We investigate the problem of multi-domain Dialogue State Tracking (DST) with open vocabulary. Existing approaches exploit BERT encoder and copy-based RNN decoder, where the encoder predicts the state operation, and the decoder generates new slot values. However, in such a stacked encoder-decoder structure, the operation prediction objective only affects the BERT encoder and the value generation objective mainly affects the RNN decoder. In this paper, we propose a purely Transformer-based framework, where a single BERT works as both the encoder and the decoder. In so doing, the operation predi…
MediaTek Research（联发科研究院）；Imperial College London（帝国理工学院）
In a dialogue system pipeline, a natural language generation (NLG) unit converts the dialogue direction and content to a corresponding natural language realization. A recent trend for dialogue systems is to first pre-train on large datasets and then fine-tune in a supervised manner using datasets annotated with application-specific features. Though novel behaviours can be learned from custom annotation, the required effort severely bounds the quantity of the training set, and the application-specific nature limits the reuse. In light of the recent success of data-driven approaches, we propose …
