In this paper, we introduce InstructSAM, a unified and streamlined framework designed for multi-instance segmentation under arbitrary instructions. We formulates instruction-driven instance segmentation as a set-structured query prediction problem and propose an explicit reasoning-to-instance query interface that elegantly bridges a vision-language model (VLM) and SAM3. Specifically, a bank of learnable instance queries is injected into the VLM and contextualized with instruction and visual information, enabling each query to serve as an instance-aware slot. A hybrid-attention mechanism further promotes interaction among these queries, visual tokens, and instruction tokens, improving instance enumeration and reducing duplicate predictions. The resulting LLM-conditioned queries are projected into SAM3's detector query space to drive accurate multi-instance segmentation in a single forward pass. This design equips SAM3 with high-level instruction understanding, compositional reasoning, and instance-level set prediction without modifying its core architecture. To support training and evaluation, we further construct Inst2Seg, a high-quality and large-scale instruction-based instance segmentation dataset and benchmark that couples free-form instructions with instance-level masks. Extensive experiments show that only 2B-scale InstructSAM achieves strong results across complex instruction-driven and phrase-level referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3's agentic pipeline while enabling efficient single-pass multi-instance prediction.
InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition
Aerospace Information Research Institute, Chinese Academy of Sciences(航天信息研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology(哈尔滨工业大学);Shanghai Jiao Tong University(上海交通大学);University of Wollongong(沃林戈大学);SAIS, Shanghai Jiao Tong University(上海交通大学国际关系学院)
University of Alberta(阿尔伯塔大学);Tencent WeChat(腾讯微信);Northwestern University(西北大学);SUSTech(四川大学);Yale University(耶鲁大学);Utrecht University(乌得勒支大学);Dalian University of Technology(大连理工大学);SIAT, Chinese Academy of Sciences(中科院深圳先进技术研究院)
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);Youtu Lab, Tencent, P.R. China(腾讯视频实验室,中国);OpenGVLab, Shanghai AI Laboratory(上海AI实验室)
Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation
School of Instrument Science and Engineering, Southeast University(东南大学仪器科学与工程学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Sichuan University-Pittsburgh Institute, Sichuan University(四川大学匹兹堡学院);Department of Biomedical Engineering, Case Western Reserve University(凯斯西储大学生物医学工程系)
University of Tübingen(图宾根大学);Bernstein Centre for Computational Neuroscience(伯恩斯坦计算神经科学中心);Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所);Baylor College of Medicine(贝勒医学院)
LlamaSeg: Image Segmentation via Autoregressive Mask Generation
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室);Tsinghua University(清华大学);Shenzhen University(深圳大学);Meituan(美团);Division of AMC and Department of ECE, HKUST(HKUST AMC 分部和电子工程系)
MedPrompt: LLM-CNN Fusion with Weight Routing for Medical Image Segmentation and Classification
1 ,2 Department of Electrical;Electronic Engineering,\ University of Engineering;3Department of Computer Science \& Engineering,\ University of Engineering \& Technology, Rajshahi-6204, Bangladesh
Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos
University of Texas at Arlington(德克萨斯大学阿灵顿分校)
本文重点关注弱标记教学视频中的任务识别和动作分割,其中在训练期间仅提供视频级动作的有序序列。我们提出了一个双流框架,它利用语义和时间层次结构来识别教学视频中的顶级任务。此外,我们提出了一种新的自上而下的弱监督动作分割方法,其中预测任务用于约束细粒度动作序列的推理。在流行的Breakfast and Cooking 2数据集上的实验结果表明,我们的双流分层任务建模在所有数据集和指标的顶级任务识别中显着优于现有方法。此外,在所提出的自上而下的动作分割方法中使用我们的任务识别框架,始终提高了最新技术水平,同时还将分割推理时间减少了80- 90%。
Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);The Institute of Artificial Intelligence and Robotics(人工智能与机器人研究所);School of Information Science and Technology(信息科学与技术学院);Zhongguancun Academy(中关村学院)
Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
The Australian National University(澳大利亚国立大学);Mitsubishi Electric Research Labs(三菱电机研究院);Technion Israel Institute of Technology(以色列理工学院);The Australian Institute for Machine Learning(澳大利亚机器学习研究所)
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation
The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);Sensetime Research Institute(商汤研究院);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought
Northwestern University(西北大学);Northeastern University(东北大学);South China University of Technology(华南理工大学);Hong Kong Baptist University(香港 Baptist大学);Beijing Normal - Hong Kong Baptist University(北京师范大学-香港 Baptist大学)