Augmented listening devices, such as hearing aids and augmented reality headsets, enhance human perception by changing the sounds that we hear. Microphone arrays can improve the performance of listening systems in noisy environments, but most array-based listening systems are designed to isolate a single sound source from a mixture. This work considers a source-remixing filter that alters the relative level of each source independently. Remixing rather than separating sounds can help to improve perceptual transparency: it causes less distortion to the signal spectrum and especially to the interaural cues that humans use to localize sounds in space.
Deep Learning for Personalized Binaural Audio Reproduction
Lab of Artificial Intelligence for Education, East China Normal University（人工智能教育实验室，华东师范大学）；Guangdong Key Laboratory of Intelligent Information Processing, College of Electronics and Information Engineering, Shenzhen University（智能信息处理重点实验室，深圳大学电子与信息工程学院）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Electronics and Communication Engineering, Guangzhou University（电子与通信工程学院，广州大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，华东师范大学）；Medizinische Physik, Carl von Ossietzky University of Oldenburg（医学物理，奥尔登堡大学）；Institute of Acoustics, Chinese Academy of Sciences（声学研究所，中国科学院）
University of Oldenburg（奥尔登堡大学）；Starkey Hearing Technologies（斯达克听力技术公司）
Binaural beamforming algorithms for head-mounted assistive listening devices are crucial to improve speech quality and speech intelligibility in noisy environments, while maintaining the spatial impression of the acoustic scene. While the well-known BMVDR beamformer is able to preserve the binaural cues of one desired source, the BLCMV beamformer uses additional constraints to also preserve the binaural cues of interfering sources. In this paper, we provide theoretical and practical insights on how to optimally set the interference scaling parameters in the BLCMV beamformer for an arbitrary nu…
Spatial Analysis and Synthesis Methods: Subjective and Objective Evaluations Using Various Microphone Arrays in the Auralization of a Critical Listening Room
University of Huddersfield（哈德斯菲尔德大学）；Genelec OY（吉力科技公司）
University of Surrey（萨里大学）；Qingdao University of Science and Technology（青岛科技大学）
Typical methods for binaural source separation consider only the direct sound as the target signal in a mixture. However, in most scenarios, this assumption limits the source separation performance. It is well known that the early reflections interact with the direct sound, producing acoustic effects at the listening position, e.g. the so-called comb filter effect. In this article, we propose a novel source separation model, that utilizes both the direct sound and the first early reflection information to model the comb filter effect. This is done by observing the interaural phase difference o…
Remixing separated audio sources trades off interferer attenuation against the amount of audible deteriorations. This paper proposes a non-intrusive audio quality estimation method for controlling this trade-off in a signal-adaptive manner. The recently proposed 2f-model is adopted as the underlying quality measure, since it has been shown to correlate strongly with basic audio quality in source separation. An alternative operation mode of the measure is proposed, more appropriate when considering material with long inactive periods of the target source. The 2f-model requires the reference tar…
Lightweight Implicit Neural Network for Binaural Audio Synthesis
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，东华大学）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，东华大学）；Department of Big Data and Information Engineering, Guizhou Industry Polytechnic College, China（大数据与信息工程系，贵州工业职业技术学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices such as hearing aids often perform poorly in noisy and reverberant environments with many competing sound sources. Large distributed microphone arrays can improve performance, but data from remote microphones often cannot be used for delay-constrained real-time processing. We present a cooperative audio source separation and enhancement system that leverages wearable listening devices and other microphone arrays spread around a room. The full distributed array is used to separate sound sources and estimate their statistics. Each listening device uses these statistic…
Sound event detection (SED) and Acoustic scene classification (ASC) are two widely researched audio tasks that constitute an important part of research on acoustic scene analysis. Considering shared information between sound events and acoustic scenes, performing both tasks jointly is a natural part of a complex machine listening system. In this paper, we investigate the usefulness of several spatial audio features in training a joint deep neural network (DNN) model performing SED and ASC. Experiments are performed for two different datasets containing binaural recordings and synchronous sound…
Besides reducing undesired sources (interfering sources and background noise), another important objective of a binaural beamforming algorithm is to preserve the spatial impression of the acoustic scene, which can be achieved by preserving the binaural cues of all sound sources. While the binaural minimum variance distortionless response (BMVDR) beamformer provides a good noise reduction performance and preserves the binaural cues of the desired source, it does not allow to control the reduction of the interfering sources and distorts the binaural cues of the interfering sources and the backgr…
University of Engineering and Technology Peshawar（白沙瓦工程技术大学）；Qatar University（卡塔尔大学）
Reverberations are unavoidable in enclosures, resulting in reduced intelligibility for hearing impaired and non native listeners and even for the normal hearing listeners in noisy circumstances. It also degrades the performance of machine listening applications. In this paper, we propose a novel approach of binaural dereverberation of a single speech source, using the differences in the interaural cues of the direct path signal and the reverberations. Two beamformers, spaced at an interaural distance, are used to extract the reverberations from the reverberant speech. The interaural cues gener…
Audio quality assessment is critical for assessing the perceptual realism of sounds. However, the time and expense of obtaining ''gold standard'' human judgments limit the availability of such data. For AR&VR, good perceived sound quality and localizability of sources are among the key elements to ensure complete immersion of the user. Our work introduces SAQAM which uses a multi-task learning framework to assess listening quality (LQ) and spatialization quality (SQ) between any given pair of binaural signals without using any subjective data. We model LQ by training on a simulated dataset of …
Accelerating Audio Research with Robotic Dummy Heads
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Stony Brook University（纽约州立大学石溪分校）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Discovery Partners Institute（发现伙伴研究所）；Amazon Web Services（亚马逊云服务）
Many hearing-impaired listeners struggle to localize sounds due to poor availability of binaural cues. Listeners with a cochlear implant and a contralateral hearing aid -- so-called bimodal listeners -- are amongst the worst performers, as both interaural time and level differences are poorly transmitted. We present a new method to enhance head shadow in the low frequencies. Head shadow enhancement is achieved with a fixed beamformer with contralateral attenuation in each ear. The method results in interaural level differences which vary monotonically with angle. It also improves low-frequency…
