Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes.
The Ohio State University（俄亥俄州立大学）；Facebook Reality Labs（Facebook现实实验室）；Facebook AI Research（Facebook人工智能研究院）；Tel-Aviv University（特拉维夫大学）
Most existing deep learning based binaural speaker separation systems focus on producing a monaural estimate for each of the target speakers, and thus do not preserve the interaural cues, which are crucial for human listeners to perform sound localization and lateralization. In this study, we address talker-independent binaural speaker separation with interaural cues preserved in the estimated binaural signals. Specifically, we extend a newly-developed gated recurrent neural network for monaural separation by additionally incorporating self-attention mechanisms and dense connectivity. We devel…
Binaural speech separation in real-world scenarios often involves moving speakers. Most current speech separation methods use utterance-level permutation invariant training (u-PIT) for training. In inference time, however, the order of outputs can be inconsistent over time particularly in long-form speech separation. This situation which is referred to as the speaker swap problem is even more problematic when speakers constantly move in space and therefore poses a challenge for consistent placement of speakers in output channels. Here, we describe a real-time binaural speech separation model b…
University of Engineering and Technology, Peshawar（白沙瓦工程技术大学）；Qatar University（卡塔尔大学）；MCS, NUST（巴基斯坦国立科技大学军事学院）
Reverberation results in reduced intelligibility for both normal and hearing-impaired listeners. This paper presents a novel psychoacoustic approach of dereverberation of a single speech source by recycling a pre-trained binaural anechoic speech separation neural network. As training the deep neural network (DNN) is a lengthy and computationally expensive process, the advantage of using a pre-trained separation network for dereverberation is that the network does not need to be retrained, saving both time and computational resources. The interaural cues of a reverberant source are given to thi…
We investigate a speech enhancement method based on the binaural coherence-to-diffuse power ratio (CDR), which preserves auditory spatial cues for maskers and a broadside target. Conventional CDR estimators typically rely on a mathematical coherence model of the desired signal and/or diffuse noise field in their formulation, which may influence their accuracy in natural environments. This work proposes a new robust and parameterized directional binaural CDR estimator. The estimator is calculated in the time-frequency domain and is based on a geometrical interpretation of the spatial coherence …
Auditory perception involves cues in the monaural auditory pathways as well as binaural cues based on differences between the ears. So far auditory models have often focused on either monaural or binaural experiments in isolation. Although binaural models typically build upon stages of (existing) monaural models, only a few attempts have been made to extend a monaural model by a binaural stage using a unified decision stage for monaural and binaural cues. In such approaches, a typical prototype of binaural processing has been the classical equalization-cancelation mechanism, which either invol…
In this paper, a binaural beamforming algorithm for hearing aid applications is introduced.The beamforming algorithm is designed to be robust to some error in the estimate of the target speaker direction. The algorithm has two main components: a robust target linearly constrained minimum variance (TLCMV) algorithm based on imposing two constraints around the estimated direction of the target signal, and a post-processor to help with the preservation of binaural cues. The robust TLCMV provides a good level of noise reduction and low level of target distortion under realistic conditions. The pos…
Besides reducing undesired sources (interfering sources and background noise), another important objective of a binaural beamforming algorithm is to preserve the spatial impression of the acoustic scene, which can be achieved by preserving the binaural cues of all sound sources. While the binaural minimum variance distortionless response (BMVDR) beamformer provides a good noise reduction performance and preserves the binaural cues of the desired source, it does not allow to control the reduction of the interfering sources and distorts the binaural cues of the interfering sources and the backgr…
University of Oldenburg（奥尔登堡大学）；Starkey Hearing Technologies（斯达克听力技术公司）
Binaural beamforming algorithms for head-mounted assistive listening devices are crucial to improve speech quality and speech intelligibility in noisy environments, while maintaining the spatial impression of the acoustic scene. While the well-known BMVDR beamformer is able to preserve the binaural cues of one desired source, the BLCMV beamformer uses additional constraints to also preserve the binaural cues of interfering sources. In this paper, we provide theoretical and practical insights on how to optimally set the interference scaling parameters in the BLCMV beamformer for an arbitrary nu…
University of Sheffield（谢菲尔德大学）；Technical University of Denmark（丹麦技术大学）
This paper presents a novel machine-hearing system that exploits deep neural networks (DNNs) and head movements for robust binaural localisation of multiple sources in reverberant environments. DNNs are used to learn the relationship between the source azimuth and binaural cues, consisting of the complete cross-correlation function (CCF) and interaural level differences (ILDs). In contrast to many previous binaural hearing systems, the proposed approach is not restricted to localisation of sound sources in the frontal hemifield. Due to the similarity of binaural cues in the frontal and rear he…
University of Engineering and Technology Peshawar（白沙瓦工程技术大学）；Qatar University（卡塔尔大学）
Reverberations are unavoidable in enclosures, resulting in reduced intelligibility for hearing impaired and non native listeners and even for the normal hearing listeners in noisy circumstances. It also degrades the performance of machine listening applications. In this paper, we propose a novel approach of binaural dereverberation of a single speech source, using the differences in the interaural cues of the direct path signal and the reverberations. Two beamformers, spaced at an interaural distance, are used to extract the reverberations from the reverberant speech. The interaural cues gener…
A Lightweight Fourier-based Network for Binaural Speech Enhancement with Spatial Cue Preservation
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，华东师范大学，中国）；Boin Hearing Technology (Shanghai) Co., LTD, China（Boin听力技术（上海）有限公司，中国）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，华东师范大学，中国）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices, such as hearing aids and augmented reality headsets, enhance human perception by changing the sounds that we hear. Microphone arrays can improve the performance of listening systems in noisy environments, but most array-based listening systems are designed to isolate a single sound source from a mixture. This work considers a source-remixing filter that alters the relative level of each source independently. Remixing rather than separating sounds can help to improve perceptual transparency: it causes less distortion to the signal spectrum and especially to the inte…
Federal University of Technology – Paraná（巴拉那联邦技术大学）；Federal University of Santa Catarina（圣卡塔琳娜联邦大学）
The multichannel Wiener filter (MWF) and its variations have been extensively applied to binaural hearing aids. However, its major drawback is the distortion of the binaural cues of the residual noise, changing the original acoustic scenario, which is of paramount importance for hearing impaired people. The MWF-IC method was previously proposed for joint speech dereverberation and noise reduction, preserving the interaural coherence (IC) of diffuse noise fields. In this work, we propose a new variation of the MWF-IC for both speech dereverberation and noise reduction, which preserves the origi…
AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers
Southern University of Science and Technology (SUSTech)（南方科技大学）；China SUSTechChina（中国南方科技大学）；City University of Hong Kong（香港城市大学）；School of Automation and Intelligent Manufacturing（自动化与智能制造学院）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Mechanical Engineering（机械工程系）
MPI for Informatics（马克斯·普朗克信息学研究所）；ETH Zürich（苏黎世联邦理工学院）；Czech Technical University（捷克技术大学）；KU Leuven（鲁汶大学）
Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360-degree camera.…
Sound reflections and late reverberation alter energetic and binaural cues of a target source, thereby affecting it's detection in noise. Two experiments investigated detection of harmonic complex tones, centered around 500 Hz, in noise in a virtual room with different modifications of simulated room impulse responses (RIR). Stimuli were auralized using the SOFE's loudspeakers in anechoic space. The target was presented from the front or at 0$^\circ$ azimuth, while an anechoic noise masker was simultaneously presented at 0$^\circ$. In the first experiment, early reflections were progressively …
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Stanford University（斯坦福大学）；Facebook AI Research（Facebook 人工智能研究院）
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization method that draws on visual information in videos to convert their monaural (single-channel) audio to binaural audio. Whereas existing approaches leverage visual features extracted directly from video frames, our approach explicitly disentangles the geometric cues present in the visual stream to guide the learning process. In particular, we develop a multi-task framework that learns geometry-aware features for binaural a…
This work presents a noise reduction method with perceptually relevant preservation of the interaural time difference (ITD) of the residual noise in binaural hearing aids. The interaural coherence (IC) concept, previously applied to the Multichannel Wiener Filter (MWF) for preservation of the spatial subjective sensation of diffuse noise fields, is proposed here to both preserve and emphasize the ITD binaural cues of a directional acoustic noise source. It is demonstrated that the previously developed MWF-ITD technique may decrease the original IC magnitude of the processed noise, consequently…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mo…
Peking University（北京大学）；Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）
Direct-path relative transfer function (DP-RTF) refers to the ratio between the direct-path acoustic transfer functions of two microphone channels. Though DP-RTF fully encodes the sound spatial cues and serves as a reliable localization feature, it is often erroneously estimated in the presence of noise and reverberation. This paper proposes to learn DP-RTF with deep neural networks for robust binaural sound source localization. A DP-RTF learning network is designed to regress the binaural sensor signals to a real-valued representation of DP-RTF. It consists of a branched convolutional neural …
STOI-optimal masking has been previously proposed and developed for single-channel speech enhancement. In this paper, we consider the extension to the task of binaural speech enhancement in which spatial information is known to be important to speech understanding and therefore should be preserved by the enhancement processing. Masks are estimated for each of the binaural channels individually and a `better-ear listening' mask is computed by choosing the maximum of the two masks. The estimated mask is used to supply probability information about the speech presence in each time-frequency bin t…
Loss functions incorporating auditory spatial perception in deep learning -- a review
School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（本-古里安大学电气与计算机工程学院）；Audio Communication Group, Technische Universität Berlin（柏林工业大学音频通信组）
Besides suppressing all undesired sound sources, an important objective of a binaural noise reduction algorithm for hearing devices is the preservation of the binaural cues, aiming at preserving the spatial perception of the acoustic scene. A well-known binaural noise reduction algorithm is the binaural minimum variance distortionless response beamformer, which can be steered using the relative transfer function (RTF) vector of the desired source, relating the acoustic transfer functions between the desired source and all microphones to a reference microphone. In this paper, we propose a compu…
Deep Learning for Personalized Binaural Audio Reproduction
Lab of Artificial Intelligence for Education, East China Normal University（人工智能教育实验室，华东师范大学）；Guangdong Key Laboratory of Intelligent Information Processing, College of Electronics and Information Engineering, Shenzhen University（智能信息处理重点实验室，深圳大学电子与信息工程学院）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Electronics and Communication Engineering, Guangzhou University（电子与通信工程学院，广州大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，华东师范大学）；Medizinische Physik, Carl von Ossietzky University of Oldenburg（医学物理，奥尔登堡大学）；Institute of Acoustics, Chinese Academy of Sciences（声学研究所，中国科学院）
Humans can robustly recognize and localize objects by integrating visual and auditory cues. While machines are able to do the same now with images, less work has been done with sounds. This work develops an approach for dense semantic labelling of sound-making objects, purely based on binaural sounds. We propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 degree camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framewo…
Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）
We present ClearBuds, the first hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for wireless earbuds requires high-quality sound separation and background cancellation, operating in real-time and on a mobile phone. Clear-Buds bridges state-of-the-art deep learning for blind audio source separation and in-ear mobile systems by making two key technical contributions: 1) a new wireless earbud design capable of operating as a synchronized, binaural microphone array, and 2) a lightweight dual-channel spee…
Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学联合具身智能研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语言认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院（KAIST））
A new database of head-related transfer functions (HRTFs) for accurate sound source localization is presented through precise measurement and post-processing in terms of improved frequency bandwidth and causality of head-related impulse responses (HRIRs) for accurate spectral cue (SC) and interaural time difference (ITD), respectively. The improvement effects of the proposed methods on binaural sound localization cues were investigated. To achieve sufficient frequency bandwidth with a single source, a one-way sealed speaker module was designed to obtain wide band frequency response based on el…
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学联合具身智能研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语种认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
Interactive audio spatialization technology previously developed for video game authoring and rendering has evolved into an essential component of platforms enabling shared immersive virtual experiences for future co-presence, remote collaboration and entertainment applications. New wearable virtual and augmented reality displays employ real-time binaural audio computing engines rendering multiple digital objects and supporting the free navigation of networked participants or their avatars through a juxtaposition of environments, real and virtual, often referred to as the Metaverse. These appl…
Speech Separation for Hearing-Impaired Children in the Classroom
Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition, and Behaviour（大脑、认知与行为研究所）；Leiden University Medical Centre（莱顿大学医学中心）；Mortimer B. Zuckerman Mind, Brain, Behavior Institute（莫蒂默·B·祖克erman心智、大脑与行为研究所）；Columbia University（哥伦比亚大学）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿大脑与认知研究所）；Delft University of Technology（代尔夫特理工大学）
EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses
ESAT Laboratory of KU Leuven（KU莱顿大学ESAT实验室）；KU Leuven（KU莱顿大学）；Department of Electrical Engineering (ESAT)（电气工程系（ESAT））；Department of Neurosciences（神经科学系）
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science, The University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）
