Deep learning architectures have made significant progress in terms of performance in many research areas. The automatic speech recognition (ASR) field has thus benefited from these scientific and technological advances, particularly for acoustic modeling, now integrating deep neural network architectures. However, these performance gains have translated into increased complexity regarding the information learned and conveyed through these black-box architectures. Following many researches in neural networks interpretability, we propose in this article a protocol that aims to determine which and where information is located in an ASR acoustic model (AM). To do so, we propose to evaluate AM performance on a determined set of tasks using intermediate representations (here, at different layer levels). Regarding the performance variation and targeted tasks, we can emit hypothesis about which information is enhanced or perturbed at different architecture steps. Experiments are performed on both speaker verification, acoustic environment classification, gender classification, tempo-distortion detection systems and speech sentiment/emotion identification. Analysis showed that neural-based AMs hold heterogeneous information that seems surprisingly uncorrelated with phoneme recognition, such as emotion, sentiment or speaker identity. The low-level hidden layers globally appears useful for the structuring of information while the upper ones would tend to delete useless information for phoneme recognition.
An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems(粤港澳人机协同智能系统联合实验室)
Center for Language and Speech Processing(语言与语音处理中心);Human Language Technology Center of Excellence(人类语言技术卓越中心);The Johns Hopkins University(约翰斯·霍普金斯大学)
Hankel-FNO: Fast Underwater Acoustic Charting Via Physics-Encoded Fourier Neural Operator
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China(信息科学与电子工程学院,浙江大学,杭州,中国);Scripps Institution of Oceanography, University of California San Diego, La Jolla, California 92093, USA(Scripps 海洋研究所,加州大学圣地亚哥分校,La Jolla,加利福尼亚州 92093,美国)
A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong(香港中文大学电子工程系);Independent Researcher(独立研究者);School of Data Science, The Chinese University of Hong Kong, Shenzhen, China(香港中文大学深圳学校数据科学系)
Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings
Centre of Mathematics and Computer Science, Stochastics group(数学与计算机科学中心,随机组);Vrije Universiteit, Department of Mathematics(自由大学,数学系);Naturalis Biodiversity Center(自然多样性中心)