arXiv 2506.042772025-06-04cs.CVcs.AI

RSVP：通过视觉提示与多模态思维链实现推理分割

RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought

Opus AI Research（Opus AI研究院）；University of Toronto（多伦多大学）；Southeast University（东南大学）；Brown University（布朗大学）；City University of Hong Kong（香港城市大学）

Yi Lu, Jiawang Cao, Yongliang Wu, Bozheng Li, Licheng Tang, Yangguang Ji, Chong Wu, Jay Wu, Wenbo Zhu

备注：: Accepted as ACL 2025 Main

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RSVP提出两阶段框架，先用多模态思维链视觉提示驱动MLLMs推理定位生成区域提议，再用视觉-语言分割模块细化生成精确掩码，在ReasonSeg和SegInW上取得领先性能。

摘要：

多模态大语言模型（MLLMs）已展现出卓越的推理能力，但缺乏显式的视觉定位与分割机制，从而在认知推理与视觉感知之间形成了鸿沟。为弥合这一鸿沟，我们提出了通过视觉提示实现推理分割的RSVP框架，这是一个将多步多模态推理与有依据的视觉理解相统一的新型框架。RSVP是一个两阶段结构化框架，将推理驱动的定位与分割细化相结合。在推理阶段，RSVP采用多模态思维链视觉提示，帮助MLLMs理解查询并推断目标，生成可解释的区域提议，从而增强视觉定位能力。在分割阶段，RSVP利用视觉-语言分割模块（VLSM）对这些提议进行细化，无缝整合文本与视觉线索以生成精确的分割掩码。通过显式建模多模态推理与分割之间的交互，RSVP为可解释的推理分割引入了一种新范式。它挖掘了MLLMs固有的定位能力，使模型不仅能够对物体进行推理，还能生成结构化的视觉表示。我们的大量实验表明，RSVP达到了最先进的性能，在ReasonSeg上超越现有最先进方法高达+6.5 gIoU和+9.2 cIoU，并在零样本设置下于SegInW上取得了49.7 mAP。这些结果验证了RSVP是一个有效且可扩展的框架，能够将认知推理与结构化视觉理解相整合。

英文摘要：

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable reasoning capability while lack explicit mechanisms for visual grounding and segmentation, creating a gap between cognitive reasoning and visual perception. To bridge this gap, we introduce Reasoning Segmentation via Visual Prompting (RSVP), a novel framework that unifies multi-step multimodal reasoning with grounded visual understanding. RSVP is a two-stage structuralized framework that integrates reasoning-driven localization with segmentation refinement. In the reasoning stage, RSVP employs multimodal chain-of-thought visual prompts to help MLLMs understand queries and infer targets, generating interpretable region proposals that enhance visual grounding. In segmentation stage, RSVP refines these proposals with a Vision-Language Segmentation Module (VLSM), seamlessly integrates textual and visual cues to produce precise segmentation masks. By explicitly modelling the interaction between multimodal reasoning and segmentation, RSVP introduces a new paradigm for interpretable reasoning segmentation. It exploits MLLMs' inherent localization capabilities, enabling the models to not only reason about objects but also generate structured visual representations. Our extensive experiments demonstrate that RSVP achieves state-of-the-art performance, surpasses state-of-the-art methods by up to +6.5 gIoU and +9.2 cIoU on ReasonSeg, and achieves 49.7 mAP on SegInW under zero-shot settings. These results validate RSVP as an effective and scalable framework for integrating cognitive reasoning with structured visual understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。