Speech is a common and effective way of communication between humans, and modern consumer devices such as smartphones and home hubs are equipped with deep learning based accurate automatic speech recognition to enable natural interaction between humans and machines. Recently, researchers have demonstrated powerful attacks against machine learning models that can fool them to produceincorrect results. However, nearly all previous research in adversarial attacks has focused on image recognition and object detection models. In this short paper, we present a first of its kind demonstration of adversarial attacks against speech classification model. Our algorithm performs targeted attacks with 87% success by adding small background noise without having to know the underlying model parameter and architecture. Our attack only changes the least significant bits of a subset of audio clip samples, and the noise does not change 89% the human listener's perception of the audio clip as evaluated in our human study.
Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models
Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan；Graduate Institute of Electrical Engineering National Taiwan University Taipei, Taiwan；Artificial Intelligence Center of Research Excellence National Taiwan University Taipei, Taiwan
College of Electrical Engineering and Computer Science, National Taiwan University（台湾大学电机资讯学院）
End-to-end speech recognition systems have achieved competitive results compared to traditional systems. However, the complex transformations involved between layers given highly variable acoustic signals are hard to analyze. In this paper, we present our ASR probing model, which synthesizes speech from hidden representations of end-to-end ASR to examine the information maintain after each layer calculation. Listening to the synthesized speech, we observe gradual removal of speaker variability and noise as the layer goes deeper, which aligns with the previous studies on how deep network functi…
The rise of machine-learning systems that process sensory input has brought with it a rise in comparisons between human and machine perception. But such comparisons face a challenge: Whereas machine perception of some stimulus can often be probed through direct and explicit measures, much of human perceptual knowledge is latent, incomplete, or unavailable for explicit report. Here, we explore how this asymmetry can cause such comparisons to misestimate the overlap in human and machine perception. As a case study, we consider human perception of \textit{adversarial speech} -- synthetic audio co…
What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems
Nara Institute of Science and Technology（奈良科学技术研究所）；RIKEN（理化学研究所）；Japan National Institute of Informatics（日本信息处理学会）；Tokyo Institute of Technology（东京技术大学）
Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China（X-LANCE实验室、计算机科学学院、上海交通大学、中国）；MoE Key Lab of Artificial Intelligence, Jiangsu Key Lab of Language Computing, China（人工智能MOE重点实验室、江苏省语言计算重点实验室、中国）
University of Surrey（萨里大学）；Kingfisher Labs Ltd（翠鸟实验室有限公司）；Just Access
Transcription of legal proceedings is very important to enable access to justice. However, speech transcription is an expensive and slow process. In this paper we describe part of a combined research and industrial project for building an automated transcription tool designed specifically for the Justice sector in the UK. We explain the challenges involved in transcribing court room hearings and the Natural Language Processing (NLP) techniques we employ to tackle these challenges. We will show that fine-tuning a generic off-the-shelf pre-trained Automatic Speech Recognition (ASR) system with a…
Automated speech audiometry: Can it work using open-source pre-trained Kaldi-NL automatic speech recognition?
University Medical Center Groningen, University of Groningen（格罗宁根大学医学中心，格罗宁根大学）；W.J. Kolff Institute for Biomedical Engineering and Materials Science, University Medical Center Groningen, University of Groningen（W.J.科尔夫生物医学工程与材料科学研究所，格罗宁根大学医学中心，格罗宁根大学）；University of Twente（特文特大学）
In this paper, we propose a novel deep transfer learning method called deep implicit distribution alignment networks (DIDAN) to deal with cross-corpus speech emotion recognition (SER) problem, in which the labeled training (source) and unlabeled testing (target) speech signals come from different corpora. Specifically, DIDAN first adopts a simple deep regression network consisting of a set of convolutional and fully connected layers to directly regress the source speech spectrums into the emotional labels such that the proposed DIDAN can own the emotion discriminative ability. Then, such abili…
Millions of users share their experiences on social media sites, such as Twitter, which in turn generate valuable data for public health monitoring, digital epidemiology, and other analyses of population health at global scale. The first, critical, task for these applications is classifying whether a personal health event was mentioned, which we call the (PHM) problem. This task is challenging for many reasons, including typically short length of social media posts, inventive spelling and lexicons, and figurative language, including hyperbole using diseases like "heart attack" or "cancer" for …
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院、深圳大数据研究院）；Bytedance（字节跳动）
CHiME-9 ECHI: A Machine Learning Challenge for Enhancing Conversations to Address Hearing Impairment
University of Sheffield（谢菲尔德大学）；WS Audiology；ORCA Labs, WS Audiology（WS Audiology ORCA实验室）；Meta Reality Labs（Meta现实实验室）；South Westphalia University of Applied Sciences（南威斯特法伦应用科学大学）
Self-supervised learning (SSL) speech models, which can serve as powerful upstream models to extract meaningful speech representations, have achieved unprecedented success in speech representation learning. However, their effectiveness on non-speech datasets is relatively less explored. In this work, we propose an ensemble framework, with a combination of ensemble techniques, to fuse SSL speech models' embeddings. Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model. Ablat…
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
IIIT-Delhi（德里印度理工学院信息技术学院）；Adobe（奥多比公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）
In recent times, BERT based transformer models have become an inseparable part of the 'tech stack' of text processing models. Similar progress is being observed in the speech domain with a multitude of models observing state-of-the-art results by using audio transformer models to encode speech. This begs the question of what are these audio transformer models learning. Moreover, although the standard methodology is to choose the last layer embedding for any downstream task, but is it the optimal choice? We try to answer these questions for the two recent audio transformer models, Mockingjay an…
