视觉引导的声音源分离与视听预测编码
Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
AI总结：
提出视听预测编码（AVPC）方法，通过预测编码网络统一特征提取、融合和掩码预测，实现参数高效且性能更优的视觉引导声音源分离。
摘要：
视觉引导的声音源分离框架通常由三部分组成：视觉特征提取、多模态特征融合和声音信号处理。该领域的一个持续趋势是为提供信息的视觉引导定制视觉特征提取器，并单独设计特征融合模块，同时默认使用U-Net进行声音分析。然而，这种分而治之的范式参数效率低下，且由于联合优化和协调各种模型组件具有挑战性，可能获得次优性能。相比之下，本文提出了一种新方法，称为视听预测编码（AVPC），以参数高效且更有效的方式解决这一任务。AVPC的网络特点是使用一个简单的基于ResNet的视频分析网络来提取语义视觉特征，以及一个基于预测编码的声音分离网络，该网络可以在同一架构中提取音频特征、融合多模态信息并预测声音分离掩码。通过迭代最小化特征之间的预测误差，AVPC递归地整合音频和视觉信息，从而实现逐步改进的性能。此外，我们通过共同预测同一声源的两种视听表示，为AVPC开发了一种有效的自监督学习策略。大量评估表明，AVPC在分离乐器声音方面优于几种基线方法，同时显著减少了模型大小。代码可在以下网址获取：https://github.com/zjsong/Audio-Visual-Predictive-Coding。
英文摘要：
The framework of visually-guided sound source separation generally consists of three parts: visual feature extraction, multimodal feature fusion, and sound signal processing. An ongoing trend in this field has been to tailor involved visual feature extractor for informative visual guidance and separately devise module for feature fusion, while utilizing U-Net by default for sound analysis. However, such divide-and-conquer paradigm is parameter inefficient and, meanwhile, may obtain suboptimal performance as jointly optimizing and harmonizing various model components is challengeable. By contrast, this paper presents a novel approach, dubbed audio-visual predictive coding (AVPC), to tackle this task in a parameter efficient and more effective manner. The network of AVPC features a simple ResNet-based video analysis network for deriving semantic visual features, and a predictive coding-based sound separation network that can extract audio features, fuse multimodal information, and predict sound separation masks in the same architecture. By iteratively minimizing the prediction error between features, AVPC integrates audio and visual information recursively, leading to progressively improved performance. In addition, we develop a valid self-supervised learning strategy for AVPC via co-predicting two audio-visual representations of the same sound source. Extensive evaluations demonstrate that AVPC outperforms several baselines in separating musical instrument sounds, while reducing the model size significantly. Code is available at: https://github.com/zjsong/Audio-Visual-Predictive-Coding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
