Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.
Dialogue systems have the potential to change how people interact with machines but are highly dependent on the quality of the data used to train them. It is therefore important to develop good dialogue annotation tools which can improve the speed and quality of dialogue data annotation. With this in mind, we introduce LIDA, an annotation tool designed specifically for conversation data. As far as we know, LIDA is the first dialogue annotation system that handles the entire dialogue annotation pipeline from raw text, as may be the output of transcription services, to structured conversation da…
Current end-to-end retrieval-based dialogue systems are mainly based on Recurrent Neural Networks or Transformers with attention mechanisms. Although promising results have been achieved, these models often suffer from slow inference or huge number of parameters. In this paper, we propose a novel lightweight fully convolutional architecture, called DialogConv, for response selection. DialogConv is exclusively built on top of convolution to extract matching features of context and response. Dialogues are modeled in 3D views, where DialogConv performs convolution operations on embedding view, wo…
National University of Singapore（新加坡国立大学）；Universidad Politécnica de Madrid（马德里理工大学）；Robert Bosch (SEA) Pte Ltd（博世（东南亚）私人有限公司）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat, exchange of information, and grounding in emotional situations. To effectively measure the quality of such conversational agents, a model-based automatic dialogue evaluation metric (ADEM) is expected to perform well across multiple domains. Despite significant progress, an ADEM that works well in one domain does not necessarily generalize to another. This calls for a dedicated network architecture for domain generalization. To tackle the multi-domain dialogue evaluation task, we propose a Panel of Expe…
Southeast University（东南大学）；Ulm University（乌尔姆大学）；Mercedes-Benz Research & Development（梅赛德斯-奔驰研发中心）
This paper introduces a natural language understanding (NLU) framework for argumentative dialogue systems in the information-seeking and opinion building domain. The proposed framework consists of two sub-models, namely intent classifier and argument similarity. Intent classifier model stacks BiLSTM with attention mechanism on top of the pre-trained BERT model and fine-tune the model for recognizing the user intent, whereas the argument similarity model employs BERT+BiLSTM for identifying system arguments the user refers to in his or her natural language utterances. Our model is evaluated in a…
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；NVIDIA（英伟达）
Conversation designers continue to face significant obstacles when creating production quality task-oriented dialogue systems. The complexity and cost involved in schema development and data collection is often a major barrier for such designers, limiting their ability to create natural, user-friendly experiences. We frame the classification of user intent as the generation of a canonical form, a lightweight semantic representation using natural language. We show that canonical forms offer a promising alternative to traditional methods for intent classification. By tuning soft prompts for a fr…
When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue
College of Engineering and Computer Science, University of Central Florida（中佛罗里达大学工程与计算机科学学院）；Department of Marketing, University of Central Florida（中佛罗里达大学市场营销系）
Helmholtz-Zentrum Geesthacht（亥姆霍兹盖斯特哈赫特研究中心）；EPFL（洛桑联邦理工学院）；byrd valley
The learning of a new language remains to this date a cognitive task that requires considerable diligence and willpower, recent advances and tools notwithstanding. In this paper, we propose Broccoli, a new paradigm aimed at reducing the required effort by seamlessly embedding vocabulary learning into users' everyday information diets. This is achieved by inconspicuously switching chosen words encountered by the user for their translation in the target language. Thus, by seeing words in context, the user can assimilate new vocabulary without much conscious effort. We validate our approach in a …
University of California Santa Barbara（加州大学圣巴巴拉分校）；Alibaba DAMO Academy（阿里巴巴达摩院）；Tencent AI Lab（腾讯人工智能实验室）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we propose to formulate the task-oriented dialogue system as the purely natural language generation task, so as to fully leverage the large-scale pre-trained models like GPT-2 and simplify complicated delexicalization prepossessing. However, directly applying this method heavily suffers from the dialogue entity inconsistency caused by the removal of delexicalized tokens, as well as the catastrophic forgetting problem of the pre-trained model during fine-tuning, leading to unsatisfactory performance. To alleviate these problems, we design a novel GPT-Adapter-CopyNet network, whic…
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；Borealis AI
Open-domain dialogue systems aim to interact with humans through natural language texts in an open-ended fashion. Despite the recent success of super large dialogue systems such as ChatGPT, using medium-to-small-sized dialogue systems remains the common practice as they are more lightweight and accessible; however, generating diverse dialogue responses is challenging, especially with smaller models. In this work, we propose an Equal-size Hard Expectation--Maximization (EqHard-EM) algorithm to train a multi-decoder model for diverse dialogue generation. Our algorithm assigns a sample to a decod…
Context-Driven Incremental Compression for Multi-Turn Dialogue Generation
The Hong Kong University of Science and Technology（香港科技大学）；NVIDIA AI Technology Center（NVIDIA AI技术中心）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）
Inside Out: Evolving User-Centric Core Memory Trees for Long-Term Personalized Dialogue Systems
School of Information, Renmin University of China（中国人民大学信息学院）；MemTensor (Shanghai) Technology Co., Ltd.（MemTensor（上海）科技有限公司）；Institute for Advanced Algorithms Research, Shanghai（上海先进算法研究所）；China Telecom Research Institute（中国电信研究院）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；Nankai University（南开大学）
Samsung R&D Institute Bangalore（三星 Bangalore 研发研究所）
Intent detection is a crucial task in any Natural Language Understanding (NLU) system and forms the foundation of a task-oriented dialogue system. To build high-quality real-world conversational solutions for edge devices, there is a need for deploying intent detection model on device. This necessitates a light-weight, fast, and accurate model that can perform efficiently in a resource-constrained environment. To this end, we propose LIDSNet, a novel lightweight on-device intent detection model, which accurately predicts the message intent by utilizing a Deep Siamese Network for learning bette…
DROID: Dual Representation for Out-of-Scope Intent Detection
School of Business, Maynooth University（梅诺特大学商学院）；Technological University Dublin（都柏林技术大学）；Huawei Research Centre（华为研究中心）；Amazon Alexa AI（亚马逊Alexa人工智能）
How Far Can You Get Without a GPU? A Systematic Benchmark of Lightweight Hallucination Detection Across Question Answering, Dialogue, and Summarisation
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, AAIS, Peking University（北京大学人工智能研究院数据科学中心）；Tencent AI Lab（腾讯人工智能实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
With the availability of massive general-domain dialogue data, pre-trained dialogue generation appears to be super appealing to transfer knowledge from the general domain to downstream applications. In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability. This paper proposes a novel dialogue generation model with a latent structure that is easily transferable from the general domain to downstream tasks in a lightweight and t…
School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院）
With the development of deep learning, advanced dialogue generation methods usually require a greater amount of computational resources. One promising approach to obtaining a high-performance and lightweight model is knowledge distillation, which relies heavily on the pre-trained powerful teacher. Collaborative learning, also known as online knowledge distillation, is an effective way to conduct one-stage group distillation in the absence of a well-trained large teacher model. However, previous work has a severe branch homogeneity problem due to the same training objective and the independent …
EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records
Taihe County People’s Hospital（太和县人民医院）；Innovation + Research Institute（创新+研究院）；GuoChuang Cloud Technology Ltd（国创云科技有限公司）；Xunfei Healthcare Technology Ltd（讯飞医疗科技有限公司）；University of Southern California（南加州大学）；The First Affiliated Hospital of Anhui Medical University（安徽医科大学第一附属医院）
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术研究所）
Multilingual spoken dialogue systems have gained prominence in the recent past necessitating the requirement for a front-end Language Identification (LID) system. Most of the existing LID systems rely on modeling the language discriminative information from low-level acoustic features. Due to the variabilities of speech (speaker and emotional variabilities, etc.), large-scale LID systems developed using low-level acoustic features suffer from a degradation in the performance. In this approach, we have attempted to model the higher level language discriminative phonotactic information for devel…
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（教育部下一代智能搜索与推荐工程研究中心）；DIRO, Université de Montréal（蒙特利尔大学计算机科学与运筹学系）；School of Information, Renmin University of China（中国人民大学信息学院）；Xidian University（西安电子科技大学）
To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2.0, focusing on the use of pre-trained language models (PLMs). To be comprehensive, our library covers $13$ common text generation tasks and their corresponding $83$ datasets and further incorporates $45$ PLMs covering general, translation, Chinese, dialogue, controllable, distilled, prompting, and lightweight PLMs. We also implement $4$ efficient training strategies and provide $4$ generation objectives for pre-training new PLMs from scratch. To be unified, we design the interfaces to …
Dynamic Personality Adaptation in Large Language Models via State Machines
Institute for Geoinformatics, University of Münster（地理信息研究所，穆尔斯特大学）；Faculty of Mathematics and Computer Science, University of Münster（数学与计算机科学学院，穆尔斯特大学）；Department of Psychology, University of Münster（心理学系，穆尔斯特大学）；Institute of Medical Education and Student Affairs, University of Münster（医学教育与学生事务研究所，穆尔斯特大学）
