While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.
Renmin University of China（中国人民大学）；Baidu Inc.（百度公司）；Shanghai Jiao Tong University（上海交通大学）
Discriminatively localizing sounding objects in cocktail-party, i.e., mixed sound scenes, is commonplace for humans, but still challenging for machines. In this paper, we propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes. Then, class-aware object localization maps are generated in the cocktail-party scenarios by referring the pre-learned object knowledge, and the sounding objects are accordingly se…
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
University of Technology Sydney（悉尼科技大学）；The University of Queensland（昆士兰大学）；Futureverse；The Hong Kong University of Science and Technology（香港科技大学）；CSIRO DATA61（联邦科学与工业研究组织数据61研究所）；Netease Fuxi AI Lab（网易伏羲人工智能实验室）
Although several research works have been reported on audio-visual sound source localization in unconstrained videos, no datasets and metrics have been proposed in the literature to quantitatively evaluate its performance. Defining the ground truth for sound source localization is difficult, because the location where the sound is produced is not limited to the range of the source object, but the vibrations propagate and spread through the surrounding objects. Therefore we propose a new concept, Sounding Object, to reduce the ambiguity of the visual location of sound, making it possible to ann…
University of Rochester（罗切斯特大学）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）
There are rich synchronized audio and visual events in our daily life. Inside the events, audio scenes are associated with the corresponding visual objects; meanwhile, sounding objects can indicate and help to separate their individual sounds in the audio track. Based on this observation, in this paper, we propose a cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and audio-visual sound separation in a unified framework. Concretely, we can leverage grounded object-sound relations to improve the results of sound separation. Meanwhile, benefiting from di…
Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）；Alibaba Group（阿里巴巴集团）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Can Textual Semantics Mitigate Sounding Object Segmentation Preference?
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Edinburgh（爱丁堡大学）
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects. To this end, we propose a space-time memory network for sounding object localization in videos. It can simultaneously learn spatio-temporal attention over both uni-modal and cross-modal representations from audio and visual modalities. We show and analyze both quantitatively and qualitatively the effectiveness of incorporating spatio-temporal learning in localizing audio-visual objects. We demonstrate that our approach generalizes over various complex…
Unsupervised audio-visual source localization aims at localizing visible sound sources in a video without relying on ground-truth localization for training. Previous works often seek high audio-visual similarities for likely positive (sounding) regions and low similarities for likely negative regions. However, accurately distinguishing between sounding and non-sounding regions is challenging without manual annotations. In this work, we propose a simple yet effective approach for Easy Visual Sound Localization, namely EZ-VSL, without relying on the construction of positive and/or negative regio…
Tufts University（塔夫茨大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about obje…
National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）；Waseda University（早稻田大学）
Detecting sound source objects within visual observation is important for autonomous robots to comprehend surrounding environments. Since sounding objects have a large variety with different appearances in our living environments, labeling all sounding objects is impossible in practice. This calls for self-supervised learning which does not require manual labeling. Most of conventional self-supervised learning uses monaural audio signals and images and cannot distinguish sound source objects having similar appearances due to poor spatial information in audio signals. To solve this problem, thi…
MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation
Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Adelaide（阿德莱德大学）
National Yang Ming Chiao Tung University（国立阳明交通大学）；University of California Merced（加州大学默塞德分校）；Snap Inc.（Snap公司）；Google Research（谷歌研究院）
Sound localization aims to find the source of the audio signal in the visual scene. However, it is labor-intensive to annotate the correlations between the signals sampled from the audio and visual modalities, thus making it difficult to supervise the learning of a machine for this task. In this work, we propose an iterative contrastive learning framework that requires no data annotations. At each iteration, the proposed method takes the 1) localization results in images predicted in the previous iteration, and 2) semantic relationships inferred from the audio signals as the pseudo-labels. We …
Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
The University of Queensland（昆士兰大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Matrix Verse AI（矩阵 Verse 人工智能公司）；CSIRO Data61（联邦科学与工业研究组织 Data61 部门）；Macau University of Science and Technology（澳门科技大学）
UT Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）
Recent work on audio-visual navigation assumes a constantly-sounding target and restricts the role of audio to signaling the target's position. We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning (e.g., toilet flushing, door creaking) and acoustic events are sporadic or short in duration. We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target. Our model's persistent multimodal memory e…
Carnegie Mellon University（卡内基梅隆大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video. Previous single-source methods mainly used the audio-visual association as clues to localize sounding objects in each image. Due to the mixed property of multiple sound sources in the original space, there exist rare multi-source approaches to localizing multiple sources simultaneously, except for one recent work using a contrastive random walk in the graph with images and separated sound as nodes. Despite their promising performance, they can only handle a fixed number of source…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Stanford University（斯坦福大学）；Facebook AI Research（Facebook 人工智能研究院）
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization method that draws on visual information in videos to convert their monaural (single-channel) audio to binaural audio. Whereas existing approaches leverage visual features extracted directly from video frames, our approach explicitly disentangles the geometric cues present in the visual stream to guide the learning process. In particular, we develop a multi-task framework that learns geometry-aware features for binaural a…
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Mitsubishi Electric Research Labs（三菱电机研究实验室）
There exists an unequivocal distinction between the sound produced by a static source and that produced by a moving one, especially when the source moves towards or away from the microphone. In this paper, we propose to use this connection between audio and visual dynamics for solving two challenging tasks simultaneously, namely: (i) separating audio sources from a mixture using visual cues, and (ii) predicting the 3D visual motion of a sounding source using its separated audio. Towards this end, we present Audio Separator and Motion Predictor (ASMP) -- a deep learning framework that leverages…
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics
The University of Queensland（昆士兰大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Matrix Verse AI；CSIRO Data61（联邦科学与工业研究组织数据61部）；Macau University of Science and Technology（澳门科技大学）
University of Rochester（罗切斯特大学）；Meta Reality Labs Research（Meta现实实验室）
Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with multisensory inputs from an egocentric perspective. In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention. To address the first problem, we propose a geometry-aware temporal aggregation module to…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science, The University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）
The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）
The task of audio-visual sound source localization has been well studied under constrained scenes, where the audio recordings are clean. However, in real-world scenarios, audios are usually contaminated by off-screen sound and background noise. They will interfere with the procedure of identifying desired sources and building visual-sound connections, making previous studies non-applicable. In this work, we propose the Interference Eraser (IEr) framework, which tackles the problem of audio-visual sound source localization in the wild. The key idea is to eliminate the interference by redefining…
Hefei University of Technology（合肥工业大学）；SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Australian National University（澳大利亚国立大学）；Beihang University（北京航空航天大学）；NVIDIA（英伟达）
We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we …
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Xinjiang University（新疆大学）；JD Explore Academy, JD.com（京东探索研究院）
Audio-visual navigation task requires an agent to find a sound source in a realistic, unmapped 3D environment by utilizing egocentric audio-visual observations. Existing audio-visual navigation works assume a clean environment that solely contains the target sound, which, however, would not be suitable in most real-world applications due to the unexpected sound noise or intentional interference. In this work, we design an acoustically complex environment in which, besides the target sound, there exists a sound attacker playing a zero-sum game with the agent. More specifically, the attacker can…
Carnegie Mellon University（卡内基梅隆大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Audio-visual source localization is a challenging task that aims to predict the location of visual sound sources in a video. Since collecting ground-truth annotations of sounding objects can be costly, a plethora of weakly-supervised localization methods that can learn from datasets with no bounding-box annotations have been proposed in recent years, by leveraging the natural co-occurrence of audio and visual signals. Despite significant interest, popular evaluation protocols have two major flaws. First, they allow for the use of a fully annotated dataset to perform early stopping, thus signif…
Australian National University（澳大利亚国立大学）；OPPO Research Institute（OPPO研究院）；The University of Oxford（牛津大学）；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）
Self-supervised audio-visual source localization aims to locate sound-source objects in video frames without extra annotations. Recent methods often approach this goal with the help of contrastive learning, which assumes only the audio and visual contents from the same video are positive samples for each other. However, this assumption would suffer from false negative samples in real-world training. For example, for an audio sample, treating the frames from the same audio class as negative samples may mislead the model and therefore harm the learned representations e.g., the audio of a siren w…
College of Informatics, Huazhong Agricultural University, Wuhan, China（华中农业大学信息学院）；School of Computing, National University of Singapore, Singapore（新加坡国立大学计算机学院）；School of Computer Science, Adelaide University, Australia（阿德莱德大学计算机科学学院）；School of Engineering, University of Warwick, Coventry, UK（沃里克大学工程学院）；Zhejiang Yuexiu University, Shaoxing, China（浙江越秀大学）
Learning from Silence and Noise for Visual Sound Source Localization
Intelligent Multimodal Vision Analysis Universitat Pompeu Fabra Barcelona, Spain（智能多模态视觉分析Universitat Pompeu Fabra Barcelona）；MARL-IDM New York University, New York, USA（MARL-IDM纽约大学）
RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology（香港科技大学跨学科研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Spatial audio, which focuses on immersive 3D sound rendering, is widely applied in the acoustic industry. One of the key problems of current spatial audio rendering methods is the lack of personalization based on different anatomies of individuals, which is essential to produce accurate sound source positions. In this work, we address this problem from an interdisciplinary perspective. The rendering of spatial audio is strongly correlated with the 3D shape of human bodies, particularly ears. To this end, we propose to achieve personalized spatial audio by reconstructing 3D human ears with sing…
