Binaural audio gives the listener the feeling of being in the recording place and enhances the immersive experience if coupled with AR/VR. But the problem with binaural audio recording is that it requires a specialized setup which is not possible to fabricate within handheld devices as compared to traditional mono audio that can be recorded with a single microphone. In order to overcome this drawback, prior works have tried to uplift the mono recorded audio to binaural audio as a post processing step conditioning on the visual input. But all the prior approaches missed other most important information required for the task, i.e. distance of different sound producing objects from the recording setup. In this work, we argue that the depth map of the scene can act as a proxy for encoding distance information of objects in the scene and show that adding depth features along with image features improves the performance both qualitatively and quantitatively. We propose a novel encoder-decoder architecture, where we use a hierarchical attention mechanism to encode the image and depth feature extracted from individual transformer backbone, with audio features at each layer of the decoder.
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mo…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Stanford University（斯坦福大学）；Facebook AI Research（Facebook 人工智能研究院）
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization method that draws on visual information in videos to convert their monaural (single-channel) audio to binaural audio. Whereas existing approaches leverage visual features extracted directly from video frames, our approach explicitly disentangles the geometric cues present in the visual stream to guide the learning process. In particular, we develop a multi-task framework that learns geometry-aware features for binaural a…
Tianjin University（天津大学）；University of Maryland at College Park（马里兰大学学院公园分校）
We present a learning-based approach for generating binaural audio from mono audio using multi-task learning. Our formulation leverages additional information from two related tasks: the binaural audio generation task and the flipped audio classification task. Our learning model extracts spatialization features from the visual and audio input, predicts the left and right audio channels, and judges whether the left and right channels are flipped. First, we extract visual features using ResNet from the video frames. Next, we perform binaural audio generation and flipped audio classification usin…
School of Marine Science and Technology, Northwestern Polytechnical University（海洋科学与技术学院，西北工业大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen, China（西北工业大学深圳研发院，中国）
Spectro-Temporal Interference Confounds Phase Encoding in Spatial Audio Foundation Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Jilin University（吉林大学）；Hunan University（湖南大学）；University of Electronic Science and Technology of China（电子科技大学）
School of Marine Science and Technology, Northwestern Polytechnical University（西北工业大学海洋科学与技术学院）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Nagoya University（名古屋大学）；Meta Reality Labs Research（Meta现实实验室）
In this work, we present an end-to-end binaural speech synthesis system that combines a low-bitrate audio codec with a powerful binaural decoder that is capable of accurate speech binauralization while faithfully reconstructing environmental factors like ambient noise or reverb. The network is a modified vector-quantized variational autoencoder, trained with several carefully designed objectives, including an adversarial loss. We evaluate the proposed system on an internal binaural dataset with objective metrics and a perceptual study. Results show that the proposed approach matches the ground…
Binaural audio gives the listener an immersive experience and can enhance augmented and virtual reality. However, recording binaural audio requires specialized setup with a dummy human head having microphones in left and right ears. Such a recording setup is difficult to build and setup, therefore mono audio has become the preferred choice in common devices. To obtain the same impact as binaural audio, recent efforts have been directed towards lifting mono audio to binaural audio conditioned on the visual input from the scene. Such approaches have not used an important cue for the task: the di…
Stereophonic audio is an indispensable ingredient to enhance human auditory experience. Recent research has explored the usage of visual information as guidance to generate binaural or ambisonic audio from mono ones with stereo supervision. However, this fully supervised paradigm suffers from an inherent drawback: the recording of stereophonic audio usually requires delicate devices that are expensive for wide accessibility. To overcome this challenge, we propose to leverage the vastly available mono data to facilitate the generation of stereophonic audio. Our key observation is that the task …
End-to-end multi-channel speaker extraction and binaural speech synthesis
Laboratory of Noise and Audio Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing, China（声学研究所噪声与音频研究实验室，中国科学院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Communication University of China, Beijing, China（中国传媒大学，北京，中国）；Tianjin 712 Communication & Broadcasting Co., Ltd.（天津712通信广播有限公司）；Tianjin Tianan Borui Technology Co., Ltd.（天津天安伯瑞科技有限公司）
CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Sony Group Corporation（索尼集团）；Sony AI, Sony Group Corporation（索尼人工智能，索尼集团）
University of Science and Technology of China（中国科学技术大学）；Imperial College London（帝国理工学院）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure Speech（微软Azure语音部门）；University of Surrey（萨里大学）；South China University of Technology（华南理工大学）
Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a c…
National Taiwan University（台湾大学）；ASUS Intelligent Cloud Services（华硕智能云服务）
Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be vie…
The present document reviews the mathematics behind binaural rendering of sound fields that are available as spherical harmonic expansion coefficients. This process is also known as binaural ambisonic decoding. We highlight that the details entail some amount peculiarity so that one has to be well aware of the precise definitions that are chosen for some of the involved quantities to obtain a consistent formulation. We also discuss what sets of definitions produce ambisonic signals that are compatible with the most common software tools that are available.
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）
Multi-modality perception is essential to develop interactive intelligence. In this work, we consider a new task of visual information-infused audio inpainting, \ie synthesizing missing audio segments that correspond to their accompanying videos. We identify two key aspects for a successful inpainter: (1) It is desirable to operate on spectrograms instead of raw audios. Recent advances in deep semantic image inpainting could be leveraged to go beyond the limitations of traditional audio inpainting. (2) To synthesize visually indicated audio, a visual-audio joint feature space needs to be learn…
Dyson School of Design Engineering（戴森设计工程学院）；Imperial College London（伦敦帝国理工学院）
PlugSonic is a suite of web- and mobile-based applications for the curation and experience of binaural interactive soundscapes and sonic narratives. It was developed as part of the PLUGGY EU project (Pluggable Social Platform for Heritage Awareness and Participation) and consists of two main applications: PlugSonic Sample, to edit and apply audio effects, and PlugSonic Soundscape, to create and experience binaural soundscapes. The audio processing within PlugSonic is based on the Web Audio API and the 3D Tune-In Toolkit, while the exploration of soundscapes in a physical space is obtained usin…
Sound source proximity and distance estimation are of great interest in many practical applications, since they provide significant information for acoustic scene analysis. As both tasks share complementary qualities, ensuring efficient interaction between these two is crucial for a complete picture of an aural environment. In this paper, we aim to investigate several ways of performing joint proximity and direction estimation from binaural recordings, both defined as coarse classification problems based on Deep Neural Networks (DNNs). Considering the limitations of binaural audio, we propose …
INRIA Grenoble Rhône-Alpes（法国国家信息与自动化研究所格勒诺布尔-罗纳-阿尔卑斯分所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Technion-Israel Inst. of Technology（以色列理工学院）
This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on monaural segregation. The method starts with a training stage that establishes a locally-linear Gaussian regression model between the directional coordinates of all the sources and the auditory features extracted from binaural measurements. While fixed-length wide-spectrum sounds (w…
Telepresence aims to create an immersive but virtual experience of the audio and visual scene at the far end for users at the near end. In this contribution, we propose an array-based binaural rendering system that converts the array microphone signals into the head-related transfer function (HRTF) filtered output signals for headphone-rendering. The proposed approach is formulated in light of a model-matching principle (MMP) and is capable of delivering more immersive experience than the conventional localization-beamforming-HRTF filtering (LBH) approach. The MMP-based rendering system can be…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices, such as hearing aids and augmented reality headsets, enhance human perception by changing the sounds that we hear. Microphone arrays can improve the performance of listening systems in noisy environments, but most array-based listening systems are designed to isolate a single sound source from a mixture. This work considers a source-remixing filter that alters the relative level of each source independently. Remixing rather than separating sounds can help to improve perceptual transparency: it causes less distortion to the signal spectrum and especially to the inte…
Lightweight Implicit Neural Network for Binaural Audio Synthesis
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，东华大学）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，东华大学）；Department of Big Data and Information Engineering, Guizhou Industry Polytechnic College, China（大数据与信息工程系，贵州工业职业技术学院）
Deep Learning for Personalized Binaural Audio Reproduction
Lab of Artificial Intelligence for Education, East China Normal University（人工智能教育实验室，华东师范大学）；Guangdong Key Laboratory of Intelligent Information Processing, College of Electronics and Information Engineering, Shenzhen University（智能信息处理重点实验室，深圳大学电子与信息工程学院）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；School of Electronics and Communication Engineering, Guangzhou University（电子与通信工程学院，广州大学）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院，华东师范大学）；Medizinische Physik, Carl von Ossietzky University of Oldenburg（医学物理，奥尔登堡大学）；Institute of Acoustics, Chinese Academy of Sciences（声学研究所，中国科学院）
Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学联合具身智能研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语言认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
MPI for Informatics（马克斯·普朗克信息学研究所）；ETH Zürich（苏黎世联邦理工学院）；Czech Technical University（捷克技术大学）；KU Leuven（鲁汶大学）
Humans can robustly recognize and localize objects by using visual and/or auditory cues. While machines are able to do the same with visual data already, less work has been done with sounds. This work develops an approach for scene understanding purely based on binaural sounds. The considered tasks include predicting the semantic masks of sound-making objects, the motion of sound-making objects, and the depth map of the scene. To this aim, we propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360-degree camera.…
Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural …
In this paper, we compare the performance of using binaural audio features in place of single-channel features for sound event detection. Three different binaural features are studied and evaluated on the publicly available TUT Sound Events 2017 dataset of length 70 minutes. Sound event detection is performed separately with single-channel and binaural features using stacked convolutional and recurrent neural network and the evaluation is reported using standard metrics of error rate and F-score. The studied binaural features are seen to consistently perform equal to or better than the single-…
Inria Rennes - Bretagne Atlantique（法国国家信息与自动化研究所雷恩-布列塔尼大西洋分所）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）
This paper introduces a new framework for supervised sound source localization referred to as virtually-supervised learning. An acoustic shoe-box room simulator is used to generate a large number of binaural single-source audio scenes. These scenes are used to build a dataset of spatial binaural features annotated with acoustic properties such as the 3D source position and the walls' absorption coefficients. A probabilistic high- to low-dimensional regression framework is used to learn a mapping from these features to the acoustic properties. Results indicate that this mapping successfully est…
AudioGS: Spectrogram-Based Audio Gaussian Splatting for Sound Field Reconstruction
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；Institute of Cultural and Creative Industry, Shanghai Jiao Tong University（上海交通大学文创学院）
