arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26102cs.CV
收藏

InstructSAM: 根据任意指令分割任意实例

InstructSAM: Segment Any Instance with Any Instructions

Zhejiang University(浙江大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学)

Yuqian Yuan, Wentong Li, Zhaocheng Li, Yutong Lin, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang, Wenqiao Zhang

AI总结:

提出InstructSAM框架,通过将指令驱动实例分割建模为集合结构查询预测问题,并设计显式推理到实例查询接口,结合视觉语言模型和SAM3实现单次前向传播中的多实例分割。

摘要:

在本文中,我们介绍了InstructSAM,一个统一且精简的框架,旨在任意指令下进行多实例分割。我们将指令驱动的实例分割形式化为一个集合结构查询预测问题,并提出了一个显式的推理到实例查询接口,优雅地桥接了视觉语言模型(VLM)和SAM3。具体来说,一组可学习的实例查询被注入到VLM中,并与指令和视觉信息进行上下文关联,使每个查询成为一个实例感知槽。混合注意力机制进一步促进了这些查询、视觉令牌和指令令牌之间的交互,改进了实例枚举并减少了重复预测。得到的LLM条件查询被投影到SAM3的检测器查询空间中,以在单次前向传播中驱动准确的多实例分割。这种设计赋予了SAM3高级指令理解、组合推理和实例级集合预测的能力,而无需修改其核心架构。为了支持训练和评估,我们进一步构建了Inst2Seg,一个高质量、大规模的基于指令的实例分割数据集和基准,将自由形式的指令与实例级掩码配对。大量实验表明,仅2B规模的InstructSAM在复杂的指令驱动和短语级指代分割基准上取得了强劲的结果,超越了之前的端到端方法和SAM3的代理流水线,同时实现了高效的单次多实例预测。

英文摘要:

In this paper, we introduce InstructSAM, a unified and streamlined framework designed for multi-instance segmentation under arbitrary instructions. We formulates instruction-driven instance segmentation as a set-structured query prediction problem and propose an explicit reasoning-to-instance query interface that elegantly bridges a vision-language model (VLM) and SAM3. Specifically, a bank of learnable instance queries is injected into the VLM and contextualized with instruction and visual information, enabling each query to serve as an instance-aware slot. A hybrid-attention mechanism further promotes interaction among these queries, visual tokens, and instruction tokens, improving instance enumeration and reducing duplicate predictions. The resulting LLM-conditioned queries are projected into SAM3's detector query space to drive accurate multi-instance segmentation in a single forward pass. This design equips SAM3 with high-level instruction understanding, compositional reasoning, and instance-level set prediction without modifying its core architecture. To support training and evaluation, we further construct Inst2Seg, a high-quality and large-scale instruction-based instance segmentation dataset and benchmark that couples free-form instructions with instance-level masks. Extensive experiments show that only 2B-scale InstructSAM achieves strong results across complex instruction-driven and phrase-level referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3's agentic pipeline while enabling efficient single-pass multi-instance prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑