Most research on task oriented dialog modeling is based on written text input. However, users interact with practical dialog systems often using speech as input. Typically, systems convert speech into text using an Automatic Speech Recognition (ASR) system, introducing errors. Furthermore, these systems do not address the differences in written and spoken language. The research on this topic is stymied by the lack of a public corpus. Motivated by these considerations, our goal in hosting the speech-aware dialog state tracking challenge was to create a public corpus or task which can be used to investigate the performance gap between the written and spoken forms of input, develop models that could alleviate this gap, and establish whether Text-to-Speech-based (TTS) systems is a reasonable surrogate to the more-labor intensive human data collection. We created three spoken versions of the popular written-domain MultiWoz task -- (a) TTS-Verbatim: written user inputs were converted into speech waveforms using a TTS system, (b) Human-Verbatim: humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans paraphrased the user inputs. Additionally, we provided different forms of ASR output to encourage wider participation from teams that may not have access to state-of-the-art ASR systems. These included ASR transcripts, word time stamps, and latent representations of the audio (audio encoder outputs). In this paper, we describe the corpus, report results from participating teams, provide preliminary analyses of their results, and summarize the current state-of-the-art in this domain.
Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
Alibaba Group(阿里巴巴集团);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents
DAMO Academy, Alibaba Group(阿里巴巴集团大模型研究院);Computer Science and Engineering Division, University of Michigan(密歇根大学计算机科学与工程系);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学光明人工智能学院)
Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech
Centre for Language Studies, Radboud University, Netherlands(语言研究所以及鲁文大学,荷兰);Department of Language and Communication, Radboud University, Netherlands(语言与交流系,鲁文大学,荷兰)
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
Tencent(腾讯);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Nanyang Technological University(南洋理工大学);Shenzhen University(深圳大学)
Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);School of Data Science, University of Science and Technology of China(中国科学技术大学数据科学学院);Tencent Youtu Lab(腾讯优图实验室)