Self-supervised audio-visual learning aims to capture useful representations of video by leveraging correspondences between visual and audio inputs. Existing approaches have focused primarily on matching semantic information between the sensory streams. We propose a novel self-supervised task to leverage an orthogonal principle: matching spatial information in the audio stream to the positions of sound sources in the visual stream. Our approach is simple yet effective. We train a model to determine whether the left and right audio channels have been flipped, forcing it to reason about spatial localization across the visual and audio streams. To train and evaluate our method, we introduce a large-scale video dataset, YouTube-ASMR-300K, with spatial audio comprising over 900 hours of footage. We demonstrate that understanding spatial correspondence enables models to perform better on three audio-visual tasks, achieving quantitative gains over supervised and self-supervised baselines that do not leverage spatial audio cues. We also show how to extend our self-supervised approach to 360 degree videos with ambisonic audio.
Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment
School of Electrical Engineering, KAIST（韩国科学技术院电气工程学院）；Department of Electrical Engineering, POSTECH（浦项工科大学电气工程系）；School of Engineering and Applied Sciences at Harvard University（哈佛大学工程与应用科学学院）
Intelligent Sight and Sound: A Chronic Cancer Pain Dataset
Booz Allen Hamilton（博思艾伦汉密尔顿公司）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Center for Cancer Research, National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所癌症研究中心）
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）
海浪的撞击声，快速行驶的汽车的轰鸣声--声音传达了关于我们周围物体的重要信息。在这项工作中，我们表明，环境声音可以作为一个监督信号学习视觉模型。为了证明这一点，我们训练一个卷积神经网络来预测与视频帧相关联的声音的统计摘要。我们表明，通过这个过程，网络学习了一种表达对象和场景信息的表示。我们在几个识别任务上评估了这种表示，发现其性能与其他最先进的无监督学习方法相当。最后，我们通过可视化展示了网络学习单元，这些单元对通常与特征声音相关联的对象具有选择性。本文扩展了早期的会议论文Owens et al. 2016，并进行了额外的实验和讨论。
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
Surrey Institute for People-Centred AI (PAI)（萨里人本人工智能研究所）；University of Surrey（萨里大学）；Centre for Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）
