持续视听声音分离
Continual Audio-Visual Sound Separation
AI总结：
本文提出持续视听声音分离任务及ContAV-Sep方法，通过跨模态相似性蒸馏约束缓解灾难性遗忘，实现新类声源持续分离并保持旧类性能。
摘要：
本文提出了一项新颖的持续视听声音分离任务，旨在借助视觉引导，在保持对先前学习类别性能的同时，持续分离新类别的声源。该问题对于实际视觉引导的听觉感知至关重要，因为它能显著增强视听声音分离模型的适应性和鲁棒性，使其更适用于现实世界中频繁遇到新声源的场景。该任务本质上具有挑战性，因为我们的模型不仅需要有效利用当前任务中两种模态的信息，还必须保留旧任务中的跨模态关联，以在视听持续学习过程中缓解灾难性遗忘。为应对这些挑战，我们提出了一种名为ContAV-Sep（持续视听声音分离）的新方法。ContAV-Sep提出了一种新颖的跨模态相似性蒸馏约束（CrossSDC），以在增量任务中维持跨模态语义相似性，并保留旧模型中已获得的语义相似性知识，从而降低灾难性遗忘的风险。CrossSDC可以无缝集成到不同视听声音分离框架的训练过程中。实验表明，ContAV-Sep能有效缓解灾难性遗忘，并在视听声音分离方面相较于其他持续学习基线取得了显著更优的性能。代码可在以下网址获取：https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024。
英文摘要：
In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep (\textbf{Cont}inual \textbf{A}udio-\textbf{V}isual Sound \textbf{Sep}aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: \url{https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
