Overview of the Ninth Dialog System Technology Challenge: DSTC9
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-W… 展开作者
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatnia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, Dilek Hakkani-Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Liden, Kaili Huang, Shahin Shayandeh, Runze Liang, Baolin Peng, Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David Traum, Maxine Eskenazi, Ahmad Beirami, Eunjoon, Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, Rajen Subba
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in dialog systems, namely, 1. Task-oriented dialog Modeling with unstructured knowledge access, 2. Multi-domain task-oriented dialog, 3. Interactive evaluation of dialog, and 4. Situated interactive multi-modal dialog. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.
In previous research, knowledge-selection tasks mostly rely on language model-based methods or knowledge ranking. However, while approaches that rely on the language models take all knowledge as sequential input, knowledge does not contain sequential information in most circumstances. On the other hand, the knowledge-ranking methods leverage dialog history and each given knowledge snippet separately, but they do not consider information between knowledge snippets. In the Tenth Dialog System Technology Challenges (DSTC10), we participated in the second Knowledge-grounded Task-oriented Dialogue …
Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4
Universidad Politécnica de Madrid（马德里理工大学）；National University of Singapore（新加坡国立大学）；Tencent AI Lab（腾讯人工智能实验室）；University of Southern California（南加州大学）；New York University（纽约大学）；Carnegie Mellon University（卡内基梅隆大学）
LINKS Foundation（LINKS基金会）；Politecnico di Torino（都灵理工大学）
This paper presents our work for the ninth edition of the Dialogue System Technology Challenge (DSTC9). Our solution addresses the track number four: Simulated Interactive MultiModal Conversations. The task consists in providing an algorithm able to simulate a shopping assistant that supports the user with his/her requests. We address the task of response retrieval, that is the task of retrieving the most appropriate agent response from a pool of response candidates. Our approach makes use of a neural architecture based on transformer with a multi-attentive structure that conditions the respon…
Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12
INESC-ID；Instituto Superior Técnico - University of Lisbon（理工学院 - 里斯本大学）；Department of Technology, Operations, and Statistics, New York University（技术、运营与统计系，纽约大学）；Speech Technology and Machine Learning Group - Universidad Politécnica de Madrid（语音技术与机器学习小组 - 马德里理工大学）；Carnegie Mellon University（卡内基梅隆大学）
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）
The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategi…
This paper gives an overview of the Schema-Guided Dialogue State Tracking task of the 8th Dialogue System Technology Challenge. The goal of this task is to develop dialogue state tracking models suitable for large-scale virtual assistants, with a focus on data-efficient joint modeling across domains and zero-shot generalization to new APIs. This task provided a new dataset consisting of over 16000 dialogues in the training set spanning 16 domains to highlight these challenges, and a baseline model capable of zero-shot generalization to new APIs. Twenty-five teams participated, developing a ran…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）
Advancing the state-of-the-art in large-scale biomedical semantic indexing and question answering is the main focus of the BioASQ challenge. BioASQ organizes respective tasks where different teams develop systems that are evaluated on the same benchmark datasets that represent the real information needs of experts in the biomedical domain. This paper presents an overview of the ninth edition of the BioASQ challenge in the context of the Conference and Labs of the Evaluation Forum (CLEF) 2021. In this year, a new question answering task, named Synergy, is introduced to support researchers study…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also known as Code Intelligence (CI) and Programming Language Processing (PLP). We survey over 287 publications and present a software-engineering centered taxonomy for CI placing each of the works into one category describing how it best assists the software development cycle. Then, we overview the field of conversational assistants and their applications in…
Hithink RoyalFlush Information Network Co.,Ltd.（同花顺信息网络股份有限公司）
We present a knowledge-grounded dialog system developed for the ninth Dialog System Technology Challenge (DSTC9) Track 1 - Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access. We leverage transfer learning with existing language models to accomplish the tasks in this challenge track. Specifically, we divided the task into four sub-tasks and fine-tuned several Transformer models on each of the sub-tasks. We made additional changes that yielded gains in both performance and efficiency, including the combination of the model with traditional entity-matchin…
University of Cambridge（剑桥大学）；University of Würzburg（维尔茨堡大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；PolyAI Limited（PolyAI有限公司）
In task-oriented dialogue (ToD), a user holds a conversation with an artificial agent to complete a concrete task. Although this technology represents one of the central objectives of AI and has been the focus of ever more intense research and development efforts, it is currently limited to a few narrow domains (e.g., food ordering, ticket booking) and a handful of languages (e.g., English, Chinese). This work provides an extensive overview of existing methods and resources in multilingual ToD as an entry point to this exciting and emerging field. We find that the most critical factor preventi…
Taiwan International Graduate Program（台湾国际研究生学程）；National Tsing Hua University（台湾清华大学）；Academia Sinica（中央研究院）
We present an overview of the EmotionX 2019 Challenge, held at the 7th International Workshop on Natural Language Processing for Social Media (SocialNLP), in conjunction with IJCAI 2019. The challenge entailed predicting emotions in spoken and chat-based dialogues using augmented EmotionLines datasets. EmotionLines contains two distinct datasets: the first includes excerpts from a US-based TV sitcom episode scripts (Friends) and the second contains online chats (EmotionPush). A total of thirty-six teams registered to participate in the challenge. Eleven of the teams successfully submitted thei…
This paper summarizes our work on the first track of the ninth Dialog System Technology Challenge (DSTC 9), "Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access". The goal of the task is to generate responses to user turns in a task-oriented dialog that require knowledge from unstructured documents. The task is divided into three subtasks: detection, selection and generation. In order to be compute efficient, we formulate the selection problem in terms of hierarchical classification steps. We achieve our best results with this model. Alternatively, we e…
Software-Based Dialogue Systems: Survey, Taxonomy and Challenges
Universitat Politècnica de Catalunya (UPC)（加泰罗尼亚理工大学）；Department of Service and Information System Engineering (ESSI)（服务与信息系统工程系）；Department of Computer Science (CS)（计算机科学系）
Chalmers University of Technology（查尔姆斯理工大学）；Department of Mechanics and Maritime Sciences Chalmers University of Technology（查尔姆斯理工大学力学与海事科学系）
In this chapter, we provide a review of conversational agents (CAs), discussing chatbots, intended for casual conversation with a user, as well as task-oriented agents that generally engage in discussions intended to reach one or several specific goals, often (but not always) within a specific domain. We also consider the concept of embodied conversational agents, briefly reviewing aspects such as character animation and speech processing. The many different approaches for representing dialogue in CAs are discussed in some detail, along with methods for evaluating such agents, emphasizing the …
Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials（广东省量子工程与量子材料重点实验室）；School of Electronic Science and Engineering (School of Microelectronics), South China Normal University（华南师范大学电子科学学院（微电子学院））；North Carolina Central University（北卡罗来纳中央大学）；Georgia Institute of Technology（佐治亚理工学院）；Columbia University（哥伦比亚大学）；LMU Munich & Munich Center for Machine Learning（慕尼黑大学及慕尼黑机器学习中心）
Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges
Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；Beijing Academy of AI（北京人工智能研究院）；Zhijiang Lab（之江实验室）；The Chinese University of Hong Kong（香港中文大学）；Beijing Institute of Technology（北京理工大学）；East China University of Science and Technology（华东理工大学）
