arXiv 2409.193422024-09-28cs.CV

X-Prompt：面向视频目标分割的多模态视觉提示

X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation

Shanghai Engineering Research Center of AI & Robotics（上海人工智能与机器人工程研究中心）；Academy for Engineering & Technology, Fudan University（复旦大学工程技术研究院）；Shanghai Key Lab of Intelligent Information Processing（上海市智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机学院）

Pinxue Guo, Wanyun Li, Hao Huang, Lingyi Hong, Xinyu Zhou, Zhaoyu Chen, Jinglun Li, Kaixun Jiang, Wei Zhang, Wenqiang Zhang

备注：: ACMMM'2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通用多模态视频目标分割框架X-Prompt，通过预训练RGB基础模型并结合多模态视觉提示器与可插拔适配专家，在有限数据下高效适配RGB+X任务，超越全微调范式并达到最先进性能。

摘要：

多模态视频目标分割（VOS），包括RGB-热红外、RGB-深度和RGB-事件，因其能够应对传统VOS方法难以处理的挑战性场景（如极端光照、快速运动和背景干扰）而受到关注。现有方法通常需要为每个任务设计特定的附加分支并进行全参数微调以实现融合。然而，这种范式不仅重复耗费研究工作和硬件成本，而且在多模态标注数据有限的情况下还存在模型崩溃的风险。本文提出了一个名为X-Prompt的通用框架，适用于所有多模态视频目标分割任务，统称为RGB+X。X-Prompt框架首先利用RGB数据预训练视频目标分割基础模型，然后利用提示中的附加模态在有限数据下将其适配到下游多模态任务。在X-Prompt框架内，我们引入了多模态视觉提示器（MVP），它允许用各种模态提示基础模型以精确分割目标。我们进一步提出了多模态适配专家（MAEs），以可插拔的模态特定知识适配基础模型，而不损害其泛化能力。为评估X-Prompt框架的有效性，我们在4个基准上的3个任务中进行了大量实验。所提出的通用X-Prompt框架始终优于全微调范式，并取得了最先进的性能。代码：https://github.com/PinxueGuo/X-Prompt.git

英文摘要：

Multi-modal Video Object Segmentation (VOS), including RGB-Thermal, RGB-Depth, and RGB-Event, has garnered attention due to its capability to address challenging scenarios where traditional VOS methods struggle, such as extreme illumination, rapid motion, and background distraction. Existing approaches often involve designing specific additional branches and performing full-parameter fine-tuning for fusion in each task. However, this paradigm not only duplicates research efforts and hardware costs but also risks model collapse with the limited multi-modal annotated data. In this paper, we propose a universal framework named X-Prompt for all multi-modal video object segmentation tasks, designated as RGB+X. The X-Prompt framework first pre-trains a video object segmentation foundation model using RGB data, and then utilize the additional modality of the prompt to adapt it to downstream multi-modal tasks with limited data. Within the X-Prompt framework, we introduce the Multi-modal Visual Prompter (MVP), which allows prompting foundation model with the various modalities to segment objects precisely. We further propose the Multi-modal Adaptation Experts (MAEs) to adapt the foundation model with pluggable modality-specific knowledge without compromising the generalization capacity. To evaluate the effectiveness of the X-Prompt framework, we conduct extensive experiments on 3 tasks across 4 benchmarks. The proposed universal X-Prompt framework consistently outperforms the full fine-tuning paradigm and achieves state-of-the-art performance. Code: https://github.com/PinxueGuo/X-Prompt.git

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。