面向指代性视听分割中的全模态表达与推理
Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation
AI总结：
针对指代性视听分割在多模态整合与深度推理上的挑战，研究者构建了含8类多模态表达、侧重音频理解与复杂推理的OmniAVS数据集，并提出基于MLLM的OISA方法，实验证明其性能优于现有方法。
摘要：
指代性视听分割（RAVS）近年来取得了显著进展，但在多模态信息整合以及视听内容的深度理解与推理方面仍存在挑战。为拓展RAVS的研究边界并推动该领域的未来研究，我们提出了全模态指代性视听分割数据集（OmniAVS），这一全新数据集包含2104个视频和61095条多模态指代表达。OmniAVS具备三项核心创新：（1）8类可灵活组合文本、语音、声音与视觉线索的多模态表达；（2）重点关注对音频内容的理解，而非仅检测其是否存在；（3）表达中融入了复杂推理与世界知识。此外，我们提出了全模态指令分割助手（OISA），以应对OmniAVS中多模态推理与视听内容细粒度理解的挑战。OISA利用多模态大语言模型（MLLM）理解复杂线索并执行基于推理的分割。大量实验表明，OISA在OmniAVS上的表现优于现有方法，且在其他相关任务上也取得了具有竞争力的结果。
英文摘要：
Referring audio-visual segmentation (RAVS) has recently seen significant advancements, yet challenges remain in integrating multimodal information and deeply understanding and reasoning about audiovisual content. To extend the boundaries of RAVS and facilitate future research in this field, we propose Omnimodal Referring Audio-Visual Segmentation (OmniAVS), a new dataset containing 2,104 videos and 61,095 multimodal referring expressions. OmniAVS stands out with three key innovations: (1) 8 types of multimodal expressions that flexibly combine text, speech, sound, and visual cues; (2) an emphasis on understanding audio content beyond just detecting their presence; and (3) the inclusion of complex reasoning and world knowledge in expressions. Furthermore, we introduce Omnimodal Instructed Segmentation Assistant (OISA), to address the challenges of multimodal reasoning and fine-grained understanding of audiovisual content in OmniAVS. OISA uses MLLM to comprehend complex cues and perform reasoning-based segmentation. Extensive experiments show that OISA outperforms existing methods on OmniAVS and achieves competitive results on other related tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
