Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers
McGill University（麦吉尔大学）；Mila（米拉）；Université de Montréal（蒙特利尔大学）；Université Paul Sabatier（保罗·萨巴蒂埃大学）；IRIT（图卢兹计算机研究所）；CNRS（法国国家科学研究中心）；Avignon Université（阿维尼翁大学）
This paper introduces Timers and Such, a new open source dataset of spoken English commands for common voice control use cases involving numbers. We describe the gap in existing spoken language understanding datasets that Timers and Such fills, the design and creation of the dataset, and experiments with a number of ASR-based and end-to-end baseline models, the code for which has been made available as part of the SpeechBrain toolkit.
Amazon Alexa AI（亚马逊Alexa AI）；University of Edinburgh（爱丁堡大学）
Executable semantic parsing is the task of converting natural language utterances into logical forms that can be directly used as queries to get a response. We build a transfer learning framework for executable semantic parsing. We show that the framework is effective for Question Answering (Q&A) as well as for Spoken Language Understanding (SLU). We further investigate the case where a parser on a new domain can be learned by exploiting data on other domains, either via multi-task learning between the target domain and an auxiliary domain or via pre-training on the auxiliary domain and fine-t…
Swiggy (Bundl Technologies India Limited)（Swiggy（印度邦德尔科技有限公司））
Spoken language Identification (LID) systems are needed to identify the language(s) present in a given audio sample, and typically could be the first step in many speech processing related tasks such as automatic speech recognition (ASR). Automatic identification of the languages present in a speech signal is not only scientifically interesting, but also of practical importance in a multilingual country such as India. In many of the Indian cities, when people interact with each other, as many as three languages may get mixed. These may include the official language of that province, Hindi and …
We present a comprehensive study on building and adapting RNN transducer (RNN-T) models for spoken language understanding(SLU). These end-to-end (E2E) models are constructed in three practical settings: a case where verbatim transcripts are available, a constrained case where the only available annotations are SLU labels and their values, and a more restrictive case where transcripts are available but not corresponding audio. We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition (ASR) systems, followed by an SLU adaptation step. In settings where …
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, these systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. We build compositional end-to-end SLU systems that explicitly separate the added complexity of recognizing spoken mentions in SLU from the NLU task of sequence labeling. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality f…
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
Tencent（腾讯）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；Shenzhen University（深圳大学）
Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）；School of Data Science, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；School of Artificial Intelligence, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院）
CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
Designing Practical Models for Isolated Word Visual Speech Recognition
organization= Department of Computer Science \& Engineering, University of Ioannina , city= Ioannina , postcode= 45110 , country= Greece；organization= Department of Surveying \& Geoinformatics Engineering, University of West Attica , city= Athens , postcode= 12243 , country= Greece
Hungarian Research Centre for Linguistics（匈牙利语言研究中心）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；SpeechTex Inc.（SpeechTex 公司）
Hungarian is spoken by 15 million people, still, easily accessible Automatic Speech Recognition (ASR) benchmark datasets - especially for spontaneous speech - have been practically unavailable. In this paper, we introduce BEA-Base, a subset of the BEA spoken Hungarian database comprising mostly spontaneous speech of 140 speakers. It is built specifically to assess ASR, primarily for conversational AI applications. After defining the speech recognition subsets and task, several baselines - including classic HMM-DNN hybrid and end-to-end approaches augmented by cross-language transfer learning -…
Automated speech audiometry: Can it work using open-source pre-trained Kaldi-NL automatic speech recognition?
University Medical Center Groningen, University of Groningen（格罗宁根大学医学中心，格罗宁根大学）；W.J. Kolff Institute for Biomedical Engineering and Materials Science, University Medical Center Groningen, University of Groningen（W.J.科尔夫生物医学工程与材料科学研究所，格罗宁根大学医学中心，格罗宁根大学）；University of Twente（特文特大学）
The present study tackles the problem of automatically discovering spoken keywords from untranscribed audio archives without requiring word-by-word speech transcription by automatic speech recognition (ASR) technology. The problem is of practical significance in many applications of speech analytics, including those concerning low-resource languages, and large amount of multilingual and multi-genre data. We propose a two-stage approach, which comprises unsupervised acoustic modeling and decoding, followed by pattern mining in acoustic unit sequences. The whole process starts by deriving and mo…
University of Hamburg（汉堡大学）；Knowledge Technology, Department of Informatics, University of Hamburg（汉堡大学信息学系知识技术）
Previous work on emotion recognition demonstrated a synergistic effect of combining several modalities such as auditory, visual, and transcribed text to estimate the affective state of a speaker. Among these, the linguistic modality is crucial for the evaluation of an expressed emotion. However, manually transcribed spoken text cannot be given as input to a system practically. We argue that using ground-truth transcriptions during training and evaluation phases leads to a significant discrepancy in performance compared to real-world conditions, as the spoken text has to be recognized on the fl…
Mobvoi AI Lab（Mobvoi AI实验室）；University of Washington（华盛顿大学）
We highlight a practical yet rarely discussed problem in dialogue state tracking (DST), namely handling unknown slot values. Previous approaches generally assume predefined candidate lists and thus are not designed to output unknown values, especially when the spoken language understanding (SLU) module is absent as in many end-to-end (E2E) systems. We describe in this paper an E2E architecture based on the pointer network (PtrNet) that can effectively extract unknown slot values while still obtains state-of-the-art accuracy on the standard DSTC2 benchmark. We also provide extensive empirical e…
National Taiwan University of Science and Technology（台湾科技大学）
Recent developments in deep learning have led to a significant innovation in various classic and practical subjects, including speech recognition, computer vision, question answering, information retrieval and so on. In the context of natural language processing (NLP), language representations have shown giant successes in many downstream tasks, so the school of studies have become a major stream of research recently. Because the immenseness of multimedia data along with speech have spread around the world in our daily life, spoken document retrieval (SDR) has become an important research subj…
Characterizing Ranked Chinese Syllable-to-Character Mapping Spectrum: A Bridge Between the Spoken and Written Chinese Language
The Robert S. Boas Center for Genomics and Human Genetics（罗伯特·S·博阿斯基因组学与人类遗传学中心）；The Feinstein Institute for Medical Research（范斯坦医学研究所）；North Shore LIJ Health System（北岸长岛犹太医疗系统）
MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder
University of Toronto（多伦多大学）；University Health Network（大学健康网络）；Knovel Engineering Lab（Knovel工程实验室）；FPT University（佛邦大学）；Florida Institute of Technology（佛罗里达技术学院）；KU Leuven（鲁文大学）；VNU University of Engineering and Technology（越南工程与技术大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognition
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan（计算机科学与信息工程系，台湾国立台湾师范大学）；Institute of Computer Science, Academia Sinica, Taiwan（学术院计算机科学研究所，台湾）；United Link Co., Ltd., Taiwan（台湾联合链接有限公司）
Institute of Smart Systems；Artificial Intelligence (ISSAI),\ University, Nur-Sultan, Kazakhstan
This paper introduces a high-quality open-source speech synthesis dataset for Kazakh, a low-resource language spoken by over 13 million people worldwide. The dataset consists of about 93 hours of transcribed audio recordings spoken by two professional speakers (female and male). It is the first publicly available large-scale dataset developed to promote Kazakh text-to-speech (TTS) applications in both academia and industry. In this paper, we share our experience by describing the dataset development procedures and faced challenges, and discuss important future directions. To demonstrate the re…
SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Xiaomi Inc.（小米公司）
Practical dialog systems need to deal with various knowledge sources, noisy user expressions, and the shortage of annotated data. To better solve the above problems, we propose CGoDial, new challenging and comprehensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions and 574,949 dialog turns totally, covering three datasets with different knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with tree-formed knowledge, and a retrieval-based dialog (RBD) dataset with candid…
TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild
Massachusetts Institute of Technology, USA（麻省理工学院）；National Taiwan University, Taipei, Taiwan（国立台湾大学）；National Taiwan University Artificial Intelligence Center of Research Excellence, Taipei, Taiwan（国立台湾大学人工智能研究中心）；Academia Sinica, Taiwan（台湾“中央”研究院）；National Yang Ming Chiao Tung University, Taiwan（阳明交通大学）；Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, USA（信号分析与解释实验室（SAIL），南加州大学）
West African Pidgin English is a language that is significantly spoken in West Africa, consisting of at least 75 million speakers. Nevertheless, proper machine translation systems and relevant NLP datasets for pidgin English are virtually absent. In this work, we develop techniques targeted at bridging the gap between Pidgin English and English in the context of natural language generation. %As a proof of concept, we explore the proposed techniques in the area of data-to-text generation. By building upon the previously released monolingual Pidgin English text and parallel English data-to-text …
