The performance of speech enhancement algorithms in a multi-speaker scenario depends on correctly identifying the target speaker to be enhanced. Auditory attention decoding (AAD) methods allow to identify the target speaker which the listener is attending to from single-trial EEG recordings. Aiming at enhancing the target speaker and suppressing interfering speakers, reverberation and ambient noise, in this paper we propose a cognitive-driven multi-microphone speech enhancement system, which combines a neural-network-based mask estimator, weighted minimum power distortionless response convolutional beamformers and AAD. To control the suppression of the interfering speaker, we also propose an extension incorporating an interference suppression constraint. The experimental results show that the proposed system outperforms the state-of-the-art cognitive-driven speech enhancement systems in challenging reverberant and noisy conditions.
Xiaomi Corp.（小米公司）；Xi'an University of Posts and Telecommunications（西安邮电大学）；Northwestern Polytechnical University（西北工业大学）；Peng Cheng Laboratory（鹏城实验室）
Electroencephalography (EEG) plays a vital role in detecting how brain responses to different stimulus. In this paper, we propose a novel Shallow-Deep Attention-based Network (SDANet) to classify the correct auditory stimulus evoking the EEG signal. It adopts the Attention-based Correlation Module (ACM) to discover the connection between auditory speech and EEG from global aspect, and the Shallow-Deep Similarity Classification Module (SDSCM) to decide the classification result via the embeddings learned from the shallow and deep layers. Moreover, various training strategies and data augmentati…
The auditory attention decoding (AAD) approach was proposed to determine the identity of the attended talker in a multi-talker scenario by analyzing electroencephalography (EEG) data. Although the linear model-based method has been widely used in AAD, the linear assumption was considered oversimplified and the decoding accuracy remained lower for shorter decoding windows. Recently, nonlinear models based on deep neural networks (DNN) have been proposed to solve this problem. However, these models did not fully utilize both the spatial and temporal features of EEG, and the interpretability of D…
The use of electroencephalogram (EEG) as the main input signal in brain-machine interfaces has been widely proposed due to the non-invasive nature of the EEG. Here we are specifically interested in interfaces that extract information from the auditory system and more specifically in the task of classifying heard speech from EEGs. To do so, we propose to limit the preprocessing of the EEGs and use machine learning approaches to automatically extract their meaningful characteristics. More specifically, we use a regulated recurrent neural network (RNN) reservoir, which has been shown to outperfor…
AADNet: Exploring EEG Spatiotemporal Information for Fast and Accurate Orientation and Timbre Detection of Auditory Attention Based on A Cue-Masked Paradigm
West China Hospital, Sichuan University（四川大学华西医院）；Institute of Electronic Engineering, China Academy of Engineering Physics（中国工程物理研究院电子工程研究所）
Auditory Attention Decoding without Spatial Information: A Diotic EEG Study
Department of Electronic and Information Engineering, School of Engineering, The University of Osaka, Suita, Japan（大阪大学工学部电子信息工学科）；Graduate School of Engineering, The University of Osaka, Suita, Japan（大阪大学工学研究院）
Identifying the target speaker in hearing aid applications is crucial to improve speech understanding. Recent advances in electroencephalography (EEG) have shown that it is possible to identify the target speaker from single-trial EEG recordings using auditory attention decoding (AAD) methods. AAD methods reconstruct the attended speech envelope from EEG recordings, based on a linear least-squares cost function or non-linear neural networks, and then directly compare the reconstructed envelope with the speech envelopes of speakers to identify the attended speaker using Pearson correlation coef…
Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multi-speaker scenario. It has been recently shown that we can quantitatively evaluate the segregation capability by modelling the relationship between the speech signals present in an auditory scene and the cortical signals of the listener measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids whereby the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are …
Eriksholm Research Centre（埃里克肖尔姆研究中心）；Linköping University（林雪平大学）；University of Stuttgart（斯图加特大学）；Technical University of Denmark（丹麦技术大学）；Oticon A/S（奥迪康公司）
Attending to the speech stream of interest in multi-talker environments can be a challenging task, particularly for listeners with hearing impairment. Research suggests that neural responses assessed with electroencephalography (EEG) are modulated by listener`s auditory attention, revealing selective neural tracking (NT) of the attended speech. NT methods mostly rely on hand-engineered acoustic and linguistic speech features to predict the neural response. Only recently, deep neural network (DNN) models without specific linguistic information have been used to extract speech features for NT, d…
MHANet: Multi-scale Hybrid Attention Network for Auditory Attention Detection
Anhui Province Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Computer Science and Technology（计算机科学与技术学院）；Anhui University（安徽大学）
ListenNet: A Lightweight Spatio-Temporal Enhancement Nested Network for Auditory Attention Detection
Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University（安徽省多模态认知计算重点实验室，计算机科学与技术学院，安徽大学）
Auditory Attention Decoding with Task-Related Multi-View Contrastive Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Tongmyong University（东明大学）
In multi speakers environments, cochlear implant (CI) users may attend to a target sound source in a different manner from the normal hearing (NH) individuals during a conversation. This study attempted to investigate the effect of conversational sound levels on the mechanisms adopted by CI and NH listeners in selective auditory attention and how it affects their daily conversation. Nine CI users (five bilateral, three unilateral, and one bimodal) and eight NH listeners participated in this study. The behavioral speech recognition scores were collected using a matrix sentences test and neural …
Improved Feature Extraction Network for Neuro-Oriented Target Speaker Extraction
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China (USTC)（中国科学技术大学）
The electroencephalography (EEG), which is one of the easiest modes of recording brain activations in a non-invasive manner, is often distorted due to recording artifacts which adversely impacts the stimulus-response analysis. The most prominent techniques thus far attempt to improve the stimulus-response correlations using linear methods. In this paper, we propose a neural network based correlation analysis framework that significantly improves over the linear methods for auditory stimuli. A deep model is proposed for intra-subject audio-EEG analysis based on directly optimizing the correlati…
End-to-End Markov State Sequence Learning for Auditory Attention Decoding
NERC-SLIP, University of Science and Technology of China（中国科学技术大学NERC-SLIP）；School of Information and Communication Engineering, Guangzhou Maritime University（广州航海学院信息与通信工程学院）；Artificial Intelligence Research Institute, iFLYTEK Company, Ltd.（科大讯飞股份有限公司人工智能研究院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）
Decoding the speech signal that a person is listening to from the human brain via electroencephalography (EEG) can help us understand how our auditory system works. Linear models have been used to reconstruct the EEG from speech or vice versa. Recently, Artificial Neural Networks (ANNs) such as Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) based architectures have outperformed linear models in modeling the relation between EEG and speech. Before attempting to use these models in real-world applications such as hearing tests or (second) language comprehension assessment w…
Frequency-Specific Neural Response and Cross-Correlation Analysis of Envelope Following Responses to Native Speech and Music Using Multichannel EEG Signals: A Case Study
Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology（电气与电子工程系，库尔纳大学工程与技术学院）；School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University（电气工程、计算与数学科学学院， Curtin 大学）
FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北脑启发智能系统重点实验室，人工智能与自动化学院，华中科技大学）
The normalization of brain recordings from multiple subjects responding to the natural stimuli is one of the key challenges in auditory neuroscience. The objective of this normalization is to transform the brain data in such a way as to remove the inter-subject redundancies and to boost the component related to the stimuli. In this paper, we propose a deep learning framework to improve the correlation of electroencephalography (EEG) data recorded from multiple subjects engaged in an audio listening task. The proposed model extends the linear multi-way canonical correlation analysis (CCA) for a…
CA-TCN: A Causal-Anticausal Temporal Convolutional Network for Direct Auditory Attention Decoding
Department of Science, Universidad Pública de Navarra (UPNA)（纳瓦拉公共大学科学系）；BCBL, Basque Center on Cognition Brain and Language（巴斯克认知脑语言中心）；Department of Statistics, Computer Sciences and Mathematics, Universidad Pública de Navarra (UPNA)（纳瓦拉公共大学统计、计算机科学与数学系）；Ikerbasque, Basque Foundation for Science（巴斯克科学基金会）
Taiyuan University of Science and Technology（太原科技大学）；University of Surrey（萨里大学）
In a recent study of auditory evoked potential (AEP) based brain-computer interface (BCI), it was shown that, with an encoder-decoder framework, it is possible to translate human neural activity to speech (T-CAS). However, current encoder-decoder-based methods achieve T-CAS often with a two-step method where the information is passed between the encoder and decoder with a shared dimension reduction vector, which may result in a loss of information. A potential approach to this problem is to design an end-to-end method by using a dual generative adversarial network (DualGAN) without dimension r…
EEG-Derived Voice Signature for Attended Speaker Detection
National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Kriston AI Lab（科斯顿人工智能实验室）；University of Bremen（不来梅大学）
AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding
Center for Ear-EEG, Department of Electrical and Computer Engineering, Aarhus University（耳电中心，电气与计算机工程系，奥胡斯大学）；Meta Reality Labs（Meta现实实验室）；Department of Electrical Engineering (ESAT), Stadius Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven and Leuven.AI - KU Leuven institute for AI（电气工程系（ESAT），动态系统、信号处理与数据分析中心，根特大学及Leuven.AI - 根特大学人工智能研究所）；Department of Neurosciences, Research Group ExpORL, Herestraat 49 box 721, B-3000 Leuven, Belgium（神经科学系，ExpORL研究组，Belgium Leuven）
EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses
ESAT Laboratory of KU Leuven（KU莱顿大学ESAT实验室）；KU Leuven（KU莱顿大学）；Department of Electrical Engineering (ESAT)（电气工程系（ESAT））；Department of Neurosciences（神经科学系）
