arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.04585cs.CV
收藏

SAM3-I: 基于指令的分割

SAM3-I: Segment Anything with Instructions

University of Alberta(阿尔伯塔大学);Tencent WeChat(腾讯微信);Northwestern University(西北大学);SUSTech(四川大学);Yale University(耶鲁大学);Utrecht University(乌得勒支大学);Dalian University of Technology(大连理工大学);SIAT, Chinese Academy of Sciences(中科院深圳先进技术研究院)

Jingjing Li, Yue Feng, Yuchen Guo, Jincai Huang, Wei Ji, Qi Bi, Yongri Piao, Miao Zhang, Xiaoqi Zhao, Qiang Chen, Shihao Zou, Huchuan Lu, Li Cheng

AI总结:

SAM3-I通过整合概念级 grounding 和指令级推理,提升分割性能,支持复杂自然语言指令。

摘要:

Segment Anything Model 3 (SAM3) 通过可提示的概念分割推进开放词汇分割,允许用户使用短名词短语 (NP) 提示分割所有与给定概念相关的实例。尽管在概念级 grounding 方面有效,现实世界交互往往涉及更丰富的自然语言指令,结合属性、关系、动作、状态或隐含推理。目前,SAM3 依赖外部多模态代理将复杂指令转换为 NP 并进行迭代掩码过滤,导致粗略表示和有限的实例特异性。在本工作中,我们提出了 SAM3-I,作为 SAM 家族的指令遵循扩展,将概念级 grounding 和指令级推理统一于单一分割框架中。基于 SAM3,SAM3-I 引入了指令感知的级联适应机制,配有专用对齐损失,逐步对齐表达指令语义与 SAM3 的视觉-语言表示,实现自然语言指令的直接解释,同时保持其强大的概念回忆能力。为实现指令遵循学习,我们引入 HMPL-Instruct,一个大规模以指令为中心的数据集,系统覆盖了分层指令语义和多样化的目标粒度。实验表明,SAM3-I 在指称和推理基于的分割中实现了有吸引力的性能,显示 SAM3 可以有效扩展以遵循复杂自然语言指令,而不牺牲其原始概念驱动的优势。代码和数据集可在 https://github.com/debby-0527/SAM3-I 上获得。

英文摘要:

Segment Anything Model 3 (SAM3) advances open-vocabulary segmentation through promptable concept segmentation, enabling users to segment all instances associated with a given concept using short noun-phrase (NP) prompts. While effective for concept-level grounding, real-world interactions often involve far richer natural-language instructions that combine attributes, relations, actions, states, or implicit reasoning. Currently, SAM3 relies on external multi-modal agents to convert complex instructions into NPs and conducts iterative mask filtering, leading to coarse representations and limited instance specificity. In this work, we present SAM3-I, an instruction-following extension of the SAM family that unifies concept-level grounding and instruction-level reasoning within a single segmentation framework. Built upon SAM3, SAM3-I introduces an instruction-aware cascaded adaptation mechanism with dedicated alignment losses that progressively aligns expressive instruction semantics with SAM3's vision-language representations, enabling direct interpretation of natural-language instructions while preserving its strong concept recall ability. To enable instruction-following learning, we introduce HMPL-Instruct, a large-scale instruction-centric dataset that systematically covers hierarchical instruction semantics and diverse target granularities. Experiments demonstrate that SAM3-I achieves appealing performance across referring and reasoning-based segmentation, showing that SAM3 can be effectively extended to follow complex natural-language instructions without sacrificing its original concept-driven strengths. Code and dataset are available at https://github.com/debby-0527/SAM3-I.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑