驯服持续音频-视觉分割中的模态纠缠
Taming Modality Entanglement in Continual Audio-Visual Segmentation
AI总结：
本文提出CAVS任务和CMR框架，通过解决多模态语义漂移和共现混淆问题，提升持续音频-视觉分割性能。
摘要：
最近，在多模态持续学习领域取得了显著进展，旨在在多模态环境下按顺序学习新任务的同时保持对先前学习任务的性能。然而，现有方法主要集中在粗粒度任务上，在细粒度持续学习设置中处理模态纠缠存在局限。为弥合这一差距，我们引入了一个新的持续音频-视觉分割（CAVS）任务，旨在通过音频引导持续分割新类别。通过全面分析，识别出两个关键挑战：1）多模态语义漂移，即在连续任务中，发声对象被标记为背景；2）共现混淆，即频繁共现的类别容易混淆。在本工作中，设计了一个基于碰撞的多模态复习（CMR）框架，以解决这些挑战。具体而言，针对多模态语义漂移，提出了一种多模态样本选择（MSS）策略，用于选择具有高模态一致性的样本进行复习。同时，针对共现混淆，设计了一种基于碰撞的样本复习（CSR）机制，允许在训练过程中增加那些易混淆类别的复习样本频率。此外，我们构建了三个音频-视觉增量场景以验证方法的有效性。全面的实验表明，我们的方法显著优于单模态持续学习方法。
英文摘要：
Recently, significant progress has been made in multi-modal continual learning, aiming to learn new tasks sequentially in multi-modal settings while preserving performance on previously learned ones. However, existing methods mainly focus on coarse-grained tasks, with limitations in addressing modality entanglement in fine-grained continual learning settings. To bridge this gap, we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to continuously segment new classes guided by audio. Through comprehensive analysis, two critical challenges are identified: 1) multi-modal semantic drift, where a sounding objects is labeled as background in sequential tasks; 2) co-occurrence confusion, where frequent co-occurring classes tend to be confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework is designed to address these challenges. Specifically, for multi-modal semantic drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select samples with high modal consistency for rehearsal. Meanwhile, for co-occurence confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed, allowing for the increase of rehearsal sample frequency of those confusable classes during training process. Moreover, we construct three audio-visual incremental scenarios to verify effectiveness of our method. Comprehensive experiments demonstrate that our method significantly outperforms single-modal continual learning methods. Code can be seen at https://github.com/cqu-student/CAVS-CMR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
