In the study of auditory attention, it has been revealed that there exists a robust correlation between attended speech and elicited neural responses, measurable through electroencephalography (EEG). Therefore, it is possible to use the attention information available within EEG signals to guide the extraction of the target speaker in a cocktail party computationally. In this paper, we present a neuro-guided speaker extraction model, i.e. NeuroSpex, using the EEG response of the listener as the sole auxiliary reference cue to extract attended speech from monaural speech mixtures. We propose a novel EEG signal encoder that captures the attention information. Additionally, we propose a cross-attention (CA) mechanism to enhance the speech feature representations, generating a speaker extraction mask. Experimental results on a publicly available dataset demonstrate that our proposed model outperforms two baseline models across various evaluation metrics.
NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Faculty of Electrical Engineering and Computing University of Zagreb（Zagreb大学电气工程与计算学院）；Faculty of Science, Radboud University（Radboud大学科学学院）
Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multi-speaker scenario. It has been recently shown that we can quantitatively evaluate the segregation capability by modelling the relationship between the speech signals present in an auditory scene and the cortical signals of the listener measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids whereby the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are …
In order for conversational AI systems to hold more natural and broad-ranging conversations, they will require much more commonsense, including the ability to identify unstated presumptions of their conversational partners. For example, in the command "If it snows at night then wake me up early because I don't want to be late for work" the speaker relies on commonsense reasoning of the listener to infer the implicit presumption that they wish to be woken only if it snows enough to cause traffic slowdowns. We consider here the problem of understanding such imprecisely stated natural language co…
Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；Centre of Speech Technology Research, University of Edinburgh（爱丁堡大学语音技术研究中心）
NeuroVoz: a Castillian Spanish corpus of parkinsonian speech
Escuela Técnica Superior de Ingenieros de Telecomunicación, Universidad Politécnica de Madrid（马德里理工大学电信工程高等技术学院）；Hospital General Universitario Gregorio Marañón（格雷戈里奥·马拉尼翁大学综合医院）
Improved Feature Extraction Network for Neuro-Oriented Target Speaker Extraction
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China (USTC)（中国科学技术大学）
Rajshahi University of Engineering & Technology (RUET)（拉杰沙希工程技术大学）
In this paper, an improved strategy for automated text dependent speaker identification system has been proposed in noisy environment. The identification process incorporates the Neuro- Genetic hybrid algorithm with cepstral based features. To remove the background noise from the source utterance, wiener filter has been used. Different speech pre-processing techniques such as start-end point detection algorithm, pre-emphasis filtering, frame blocking and windowing have been used to process the speech utterances. RCC, MFCC, MFCC, MFCC, LPC and LPCC have been used to extract the features. After …
EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses
ESAT Laboratory of KU Leuven（KU莱顿大学ESAT实验室）；KU Leuven（KU莱顿大学）；Department of Electrical Engineering (ESAT)（电气工程系（ESAT））；Department of Neurosciences（神经科学系）
University of Coimbra（科英布拉大学）；Institute of Systems and Robotics（系统与机器人研究所）
Human beings have developed fantastic abilities to integrate information from various sensory sources exploring their inherent complementarity. Perceptual capabilities are therefore heightened, enabling, for instance, the well-known "cocktail party" and McGurk effects, i.e., speech disambiguation from a panoply of sound signals. This fusion ability is also key in refining the perception of sound source location, as in distinguishing whose voice is being heard in a group conversation. Furthermore, neuroscience has successfully identified the superior colliculus region in the brain as the one re…
As a neurophysiological response to threat or adverse conditions, stress can affect cognition, emotion and behaviour with potentially detrimental effects on health in the case of sustained exposure. Since the affective content of speech is inherently modulated by an individual's physical and mental state, a substantial body of research has been devoted to the study of paralinguistic correlates of stress-inducing task load. Historically, voice stress analysis (VSA) has been conducted using conventional digital signal processing (DSP) techniques. Despite the development of modern methods based o…
Egocentric Speaker Classification in Child-Adult Dyadic Interactions: From Sensing to Computational Modeling
University of Southern California（南加州大学）；University of California, San Francisco（加州大学旧金山分校）；Signal Analysis and Interpretation Laboratory（信号分析与解释实验室）；Weill Institute for Neurosciences（魏尔神经科学研究所）
Parkinson's disease is a neurodegenerative disorder characterized by the presence of different motor impairments. Information from speech, handwriting, and gait signals have been considered to evaluate the neurological state of the patients. On the other hand, user models based on Gaussian mixture models - universal background models (GMM-UBM) and i-vectors are considered the state-of-the-art in biometric applications like speaker verification because they are able to model specific speaker traits. This study introduces the use of GMM-UBM and i-vectors to evaluate the neurological state of Par…
Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition
The Chinese University of Hong Kong（香港中文大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；National Research Council of Canada（加拿大国家研究委员会）
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Disordered speech recognition is a highly challenging task. The underlying neuro-motor conditions of people with speech disorders, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of speech required for system development. This paper investigates a set of data augmentation techniques for disordered speech recognition, including vocal tract length perturbation (VTLP), tempo perturbation and speed perturbation. Both normal and disordered speech were exploited in the augmentation process. Variability among impaired speakers in both th…
University of Augsburg（奥格斯堡大学）；Imperial College London（帝国理工学院）
N-HANS is a Python toolkit for in-the-wild audio enhancement, including speech, music, and general audio denoising, separation, and selective noise or source suppression. The functionalities are realised based on two neural network models sharing the same architecture, but trained separately. The models are comprised of stacks of residual blocks, each conditioned on additional speech or environmental noise recordings for adapting to different unseen speakers or environments in real life. In addition to a Python API, a command line interface is provided to researchers and developers, both of wh…
The Chinese University of Hong Kong（香港中文大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Automatic recognition of disordered speech remains a highly challenging task to date. The underlying neuro-motor conditions, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of impaired speech required for ASR system development. This paper presents novel variational auto-encoder generative adversarial network (VAE-GAN) based personalized disordered speech augmentation approaches that simultaneously learn to encode, generate and discriminate synthesized impaired speech. Separate latent features are derived to learn dysarthric speec…
1 College of Electronic；Optical Engineering \& College of Flexible Electronics (Future Technology), Nanjing University of Posts；Telecommunications, Nanjing, China 2 School of Artificial Intelligence, School of Data Science, SRIBD, The Chinese University of Hong Kong, Shenzhen, China
Spiking Neural Networks (SNN) are known to be very effective for neuromorphic processor implementations, achieving orders of magnitude improvements in energy efficiency and computational latency over traditional deep learning approaches. Comparable algorithmic performance was recently made possible as well with the adaptation of supervised training algorithms to the context of SNN. However, information including audio, video, and other sensor-derived data are typically encoded as real-valued signals that are not well-suited to SNN, preventing the network from leveraging spike timing informatio…
Breaking Shortcut Learning for Cross-Trial EEG-Guided Target Speech Extraction via Two-Stage Training
Department of Intelligence and Information（智能信息系）；Seoul National University（首尔国立大学）；Department of Communication Sciences and Disorders（沟通科学与障碍系）；University of Iowa（爱荷华大学）；Interdisciplinary Program in Artificial Intelligence（人工智能交叉学科项目）；Artificial Intelligence Institute（人工智能研究所）
Confidence-Based Self-Training for EMG-to-Speech: Leveraging Synthetic EMG for Robust Modeling
ETIS, CY Cergy Paris Université - ENSEA – CNRS, UMR 8051（ETIS，CY 塞尔吉巴黎大学 - ENSEA – CNRS，UMR 8051）；Institute for Infocomm Research, A STAR（信息与通信研究所以院）；IPAL (International Research Laboratory on Artificial Intelligence), CNRS（人工智能国际研究实验室，CNRS）
Automatic speaker verification systems are vulnerable to a variety of access threats, prompting research into the formulation of effective spoofing detection systems to act as a gate to filter out such spoofing attacks. This study introduces a simple attention module to infer 3-dim attention weights for the feature map in a convolutional layer, which then optimizes an energy function to determine each neuron's importance. With the advancement of both voice conversion and speech synthesis technologies, unseen spoofing attacks are constantly emerging to limit spoofing detection system performanc…
FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北脑启发智能系统重点实验室，人工智能与自动化学院，华中科技大学）
Dysarthric speech detection (DSD) systems aim to detect characteristics of the neuromotor disorder from speech. Such systems are particularly susceptible to domain mismatch where the training and testing data come from the source and target domains respectively, but the two domains may differ in terms of speech stimuli, disease etiology, etc. It is hard to acquire labelled data in the target domain, due to high costs of annotating sizeable datasets. This paper makes a first attempt to formulate cross-domain DSD as an unsupervised domain adaptation (UDA) problem. We use labelled source-domain d…
End-to-End Markov State Sequence Learning for Auditory Attention Decoding
NERC-SLIP, University of Science and Technology of China（中国科学技术大学NERC-SLIP）；School of Information and Communication Engineering, Guangzhou Maritime University（广州航海学院信息与通信工程学院）；Artificial Intelligence Research Institute, iFLYTEK Company, Ltd.（科大讯飞股份有限公司人工智能研究院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
