Identifying the target speaker in hearing aid applications is crucial to improve speech understanding. Recent advances in electroencephalography (EEG) have shown that it is possible to identify the target speaker from single-trial EEG recordings using auditory attention decoding (AAD) methods. AAD methods reconstruct the attended speech envelope from EEG recordings, based on a linear least-squares cost function or non-linear neural networks, and then directly compare the reconstructed envelope with the speech envelopes of speakers to identify the attended speaker using Pearson correlation coefficients. Since these correlation coefficients are highly fluctuating, for a reliable decoding a large correlation window is used, which causes a large processing delay. In this paper, we investigate a state-space model using correlation coefficients obtained with a small correlation window to improve the decoding performance of the linear and the non-linear AAD methods. The experimental results show that the state-space model significantly improves the decoding performance.
AbsoluteNet: A Deep Learning Neural Network to Classify Cerebral Hemodynamic Responses of Auditory Processing
1 Department of Electrical Engineering, Computer, \& Biomedical Engineering, University of Rhode Island RI, USA 2 Multimodal Functional Brain Imaging Research Lab at St. Jude Children's Research Hospital, TN, USA 3 Miner School of Computer；Information Sciences at University of massachusetts Lowell, MA, USA
Condition-Invariant fMRI Decoding of Speech Intelligibility with Deep State Space Model
1Academia Sinica, Taiwan 2Keio University, Japan 3Graduate Institute of Communication Engineering, National Taiwan University, Taiwan 4Taiwan International Graduate Program in Interdisciplinary Neuroscience, National Taiwan University, Taiwan
NTT Communication Science Laboratories, NTT Corporation（NTT公司NTT通信科学实验室）；University of Oldenburg（奥尔登堡大学）
The performance of speech enhancement algorithms in a multi-speaker scenario depends on correctly identifying the target speaker to be enhanced. Auditory attention decoding (AAD) methods allow to identify the target speaker which the listener is attending to from single-trial EEG recordings. Aiming at enhancing the target speaker and suppressing interfering speakers, reverberation and ambient noise, in this paper we propose a cognitive-driven multi-microphone speech enhancement system, which combines a neural-network-based mask estimator, weighted minimum power distortionless response convolut…
The auditory attention decoding (AAD) approach was proposed to determine the identity of the attended talker in a multi-talker scenario by analyzing electroencephalography (EEG) data. Although the linear model-based method has been widely used in AAD, the linear assumption was considered oversimplified and the decoding accuracy remained lower for shorter decoding windows. Recently, nonlinear models based on deep neural networks (DNN) have been proposed to solve this problem. However, these models did not fully utilize both the spatial and temporal features of EEG, and the interpretability of D…
National University of Singapore（新加坡国立大学）；Agency for Science, Technology and Research（新加坡科技研究局）；University of New South Wales（新南威尔士大学）
Auditory front-end is an integral part of a spiking neural network (SNN) when performing auditory cognitive tasks. It encodes the temporal dynamic stimulus, such as speech and audio, into an efficient, effective and reconstructable spike pattern to facilitate the subsequent processing. However, most of the auditory front-ends in current studies have not made use of recent findings in psychoacoustics and physiology concerning human listening. In this paper, we propose a neural encoding and decoding scheme that is optimized for speech processing. The neural encoding scheme, that we call Biologic…
AADNet: Exploring EEG Spatiotemporal Information for Fast and Accurate Orientation and Timbre Detection of Auditory Attention Based on A Cue-Masked Paradigm
West China Hospital, Sichuan University（四川大学华西医院）；Institute of Electronic Engineering, China Academy of Engineering Physics（中国工程物理研究院电子工程研究所）
People suffering from hearing impairment often have difficulties participating in conversations in so-called `cocktail party' scenarios with multiple people talking simultaneously. Although advanced algorithms exist to suppress background noise in these situations, a hearing device also needs information on which of these speakers the user actually aims to attend to. The correct (attended) speaker can then be enhanced using this information, and all other speakers can be treated as background noise. Recent neuroscientific advances have shown that it is possible to determine the focus of audito…
Auditory Attention Decoding with Task-Related Multi-View Contrastive Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）
Understanding Auditory Evoked Brain Signal via Physics-informed Embedding Network with Multi-Task Transformer
School of Mathematics, Southwest Jiaotong University（西南交通大学数学学院）；School of Computer, SWJTU-Leeds Joint School（西南交通大学-利兹学院计算机学院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Chengdu University of Technology Oxford Brookes College（成都理工大学牛津布鲁克斯学院）
Auditory Attention Decoding without Spatial Information: A Diotic EEG Study
Department of Electronic and Information Engineering, School of Engineering, The University of Osaka, Suita, Japan（大阪大学工学部电子信息工学科）；Graduate School of Engineering, The University of Osaka, Suita, Japan（大阪大学工学研究院）
FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北脑启发智能系统重点实验室，人工智能与自动化学院，华中科技大学）
Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
University of Oxford（牛津大学）；Peking University（北京大学）；Yale University（耶鲁大学）；Tencent AI Lab（腾讯人工智能实验室）
In sequence-to-sequence learning, e.g., natural language generation, the decoder relies on the attention mechanism to efficiently extract information from the encoder. While it is common practice to draw information from only the last encoder layer, recent work has proposed to use representations from different encoder layers for diversified levels of information. Nonetheless, the decoder still obtains only a single view of the source sequences, which might lead to insufficient training of the encoder layer stack due to the hierarchy bypassing problem. In this work, we propose layer-wise multi…
AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding
Center for Ear-EEG, Department of Electrical and Computer Engineering, Aarhus University（耳电中心，电气与计算机工程系，奥胡斯大学）；Meta Reality Labs（Meta现实实验室）；Department of Electrical Engineering (ESAT), Stadius Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven and Leuven.AI - KU Leuven institute for AI（电气工程系（ESAT），动态系统、信号处理与数据分析中心，根特大学及Leuven.AI - 根特大学人工智能研究所）；Department of Neurosciences, Research Group ExpORL, Herestraat 49 box 721, B-3000 Leuven, Belgium（神经科学系，ExpORL研究组，Belgium Leuven）
Auditory attention decoding (AAD) is a technique used to identify and amplify the talker that a listener is focused on in a noisy environment. This is done by comparing the listener's brainwaves to a representation of all the sound sources to find the closest match. The representation is typically the waveform or spectrogram of the sounds. The effectiveness of these representations for AAD is uncertain. In this study, we examined the use of self-supervised learned speech representation in improving the accuracy and speed of AAD. We recorded the brain activity of three subjects using invasive e…
Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multi-speaker scenario. It has been recently shown that we can quantitatively evaluate the segregation capability by modelling the relationship between the speech signals present in an auditory scene and the cortical signals of the listener measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids whereby the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are …
Recent advances in conditional recurrent language modelling have mainly focused on network architectures (e.g., attention mechanism), learning algorithms (e.g., scheduled sampling and sequence-level training) and novel applications (e.g., image/video description generation, speech recognition, etc.) On the other hand, we notice that decoding algorithms/strategies have not been investigated as much, and it has become standard to use greedy or beam search. In this paper, we propose a novel decoding strategy motivated by an earlier observation that nonlinear hidden layers of a deep neural network…
Scattering Transform for Auditory Attention Decoding
Institute for Signal Processing, University of Luebeck（信号处理研究所，吕贝克大学）；Department for Vision and Machine Learning, FPI Food Processing Innovation GmbH & Co. KG（视觉与机器学习系，FPI食品加工创新有限责任公司）；German Research Center for Artificial Intelligence (DFKI), AI for Assistive Health Technologies（德国人工智能研究中心（DFKI），辅助健康技术人工智能）
本文提出使用两层散射变换替代传统预处理方法，以提升听觉注意力解码性能，尤其在 KUL 数据集上表现显著。
ListenNet: A Lightweight Spatio-Temporal Enhancement Nested Network for Auditory Attention Detection
Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University（安徽省多模态认知计算重点实验室，计算机科学与技术学院，安徽大学）
Magnetoencephalography (MEG) Based Non-Invasive Chinese Speech Decoding
Ministry of Education Key Laboratory of Image Processing and Intelligent Control（教育部图像处理与智能控制重点实验室）；School of Artificial Intelligence and Automation（人工智能与自动化学院）；Huazhong University of Science and Technology（华中科技大学）；Department of Neurosurgery（神经外科科）；Tongji Hospital（同济医院）；Tongji Medical College（同济医学院）
Attentive listening in a multispeaker environment such as a cocktail party requires suppression of the interfering speakers and the noise around. People with normal hearing perform remarkably well in such situations. Analysis of the cortical signals using electroencephalography (EEG) has revealed that the EEG signals track the envelope of the attended speech stronger than that of the interfering speech. This has enabled the development of algorithms that can decode the selective attention of a listener in controlled experimental settings. However, often these algorithms require longer trial du…
Speaker recognition is a biometric modality that uses underlying speech information to determine the identity of the speaker. Speaker Identification (SID) under noisy conditions is one of the challenging topics in the field of speech processing, specifically when it comes to individuals with cochlear implants (CI). This study analyzes and quantifies the ability of CI-users to perform speaker identification based on direct electric auditory stimuli. CI users employ a limited number of frequency bands (8 to 22) and use electrodes to directly stimulate the Basilar Membrane/Cochlear in order to re…
The electroencephalography (EEG), which is one of the easiest modes of recording brain activations in a non-invasive manner, is often distorted due to recording artifacts which adversely impacts the stimulus-response analysis. The most prominent techniques thus far attempt to improve the stimulus-response correlations using linear methods. In this paper, we propose a neural network based correlation analysis framework that significantly improves over the linear methods for auditory stimuli. A deep model is proposed for intra-subject audio-EEG analysis based on directly optimizing the correlati…
ASR systems designed for native English (L1) usually underperform on non-native English (L2). To address this performance gap, \textbf{(i)} we extend our previous work to investigate fine-tuning of a pre-trained wav2vec 2.0 model \cite{baevski2020wav2vec,xu2021self} under a rich set of L1 and L2 training conditions. We further \textbf{(ii)} incorporate language model decoding in the ASR system, along with the fine-tuning method. Quantifying gains acquired from each of these two approaches separately and an error analysis allows us to identify different sources of improvement within our models.…
During speech perception, a listener's electroencephalogram (EEG) reflects acoustic-level processing as well as higher-level cognitive factors such as speech comprehension and attention. However, decoding speech from EEG recordings is challenging due to the low signal-to-noise ratios of EEG signals. We report on an approach developed for the ICASSP 2023 'Auditory EEG Decoding' Signal Processing Grand Challenge. A simple ensembling method is shown to considerably improve upon the baseline decoder performance. Even higher classification rates are achieved by jointly decoding the speech-evoked fr…
To undertake machine lip-reading, we try to recognise speech from a visual signal. Current work often uses viseme classification supported by language models with varying degrees of success. A few recent works suggest phoneme classification, in the right circumstances, can outperform viseme classification. In this work we present a novel two-pass method of training phoneme classifiers which uses previously trained visemes in the first pass. With our new training algorithm, we show classification performance which significantly improves on previous lip-reading results.
Improving Speech Decoding from ECoG with Self-Supervised Pretraining
Institute of Neuroinformatics（神经信息学研究所）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；School of Electrical and Computer Engineering（电气与计算机工程学院）；Purdue University（普渡大学）
