AI 中文总结
针对AVIS的模态状态变化干扰与跨模态分布差异问题，提出含ADSM与OT-MM的SAMOT框架，在AVIS基准取得最优性能。
AI 中文摘要
视听实例分割（AVIS）旨在对视频序列内发声对象进行同步分类、分割与跟踪。与视听语义分割（AVS）不同，AVIS涉及更长视频序列的实例级建模，带来两大关键挑战：（1）复杂的模态状态变化会干扰长程建模；（2）模态间存在显著的结构与分布差异，阻碍精确的实例级关联。现有方法依赖固定步长的Transformer与递归Mamba模型，缺乏对模态状态变化的适应性；此外，采用隐式匹配的方法忽略了固有分布不一致性。为解决这些问题，本文提出包含自适应动态步长调制（ADSM）与基于最优传输的匹配调制（OT-MM）的框架：ADSM利用时间变化、跨模态差异及历史上下文自适应调制Mamba步长，平衡对模态状态变化的快速响应与稳定长程建模；OT-MM将实例级跨模态匹配显式表述为熵正则化的最优传输问题，通过对数域Sinkhorn迭代求解，并结合MMD正则器强化分布级一致性。大量实验在AVIS基准上展现出最优性能（FSLA提升3.76、HOTA提升2.75、mAP提升2.58），并通过全面定性可视化验证，代码与模型可在指定URL获取。
英文摘要
Audio-Visual Instance Segmentation (AVIS) aims to simultaneously classify, segment, and track sounding objects within video sequences. Unlike Audio-Visual Semantic Segmentation (AVS), AVIS involves instance-level modeling across longer video sequences, introducing two key challenges: (1) complex modality-state changes disrupt long-range modeling, and (2) substantial structural and distributional discrepancies between modalities hinder precise instance-level association. Existing methods rely on fixed-step Transformers and recursive Mamba models, lacking adaptability to modality-state changes. In addition, methods performing implicit matching ignore the inherent distributional inconsistencies. To address these issues, we propose a framework with Adaptive Dynamic Step Modulation (ADSM) and Optimal Transport-based Matching Modulation (OT-MM). ADSM adaptively modulates Mamba step sizes using temporal variation, cross-modal discrepancy, and historical context, balancing rapid response to modality-state changes with stable long-range modeling. OT-MM explicitly formulates instance-level cross-modal matching as an entropy-regularized optimal transport problem solved via log-domain Sinkhorn iterations, and further enforces distribution-level coherence with an MMD regularizer. Extensive experiments demonstrate state-of-the-art performance on the AVIS benchmark (+3.76 FSLA, +2.75 HOTA, +2.58 mAP), verified through comprehensive qualitative visualizations. The code and model are available at https://github.com/happylife-pk/SAMOT.
CommentsAccepted by ACM MM 2026
