尽管感知系统近年来取得了显著进展,它们仍然依赖明确的人类指令或预定义类别来在执行视觉识别任务之前识别目标对象。此类系统无法主动推理和理解隐含的用户意图。在这项工作中,我们提出了一个新的分割任务——推理分割(reasoning segmentation)。该任务旨在给定复杂且隐含的查询文本时输出分割掩码。此外,我们建立了一个包含超过一千个图像-指令-掩码数据样本的基准,其中融入了复杂的推理和世界知识以用于评估。最后,我们提出了LISA:大语言指令分割助手(large Language Instructed Segmentation Assistant),它继承了多模态大语言模型(LLMs)的语言生成能力,同时也具备生成分割掩码的能力。我们用一个<SEG>标记扩展了原始词汇表,并提出了embedding-as-mask范式以解锁分割能力。值得注意的是,LISA能够处理涉及复杂推理和世界知识的情况。同时,当仅在无推理数据集上训练时,它展现出稳健的零样本能力。此外,仅用239个推理分割数据样本对模型进行微调即可进一步提升性能。定量和定性实验均表明,我们的方法有效地为多模态LLMs解锁了新的推理分割能力。代码、模型和数据可在https://github.com/dvlab-research/LISA获取。
英文摘要:
Although perception systems have made remarkable advancements in recent years, they still rely on explicit human instruction or pre-defined categories to identify the target objects before executing visual recognition tasks. Such systems cannot actively reason and comprehend implicit user intention. In this work, we propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction-mask data samples, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of multimodal Large Language Models (LLMs) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a <SEG> token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving complex reasoning and world knowledge. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation data samples results in further performance enhancement. Both quantitative and qualitative experiments show our method effectively unlocks new reasoning segmentation capabilities for multimodal LLMs. Code, models, and data are available at https://github.com/dvlab-research/LISA.
MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training
School of Aeronautics and Astronautics, Zhejiang University(浙江大学航空宇航学院);Southern Medical University(南方医科大学);School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University(浙江科技学院计算机科学与技术学院(人工智能学院));College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院)
PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学)
Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Voyager Research, Didi Chuxing
DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing Artificial Intelligence Research of IA(中国科学院自动化所南京人工智能研究院)
PRS-Med: Position Reasoning Segmentation in Medical Imaging
Aalto University(阿尔托大学);Northwestern University(西北大学);Technical University of Denmark(丹麦技术大学);Chongqing University of Posts and Telecommunications(重庆邮电大学);University of South Dakota(南达科他大学)
GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
College of Biosystems Engineering and Food Science, Zhejiang University(浙江大学生物系统工程与食品科学学院);ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学杭州国际科创中心)
POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation
Singapore University of Technology and Design(新加坡科技设计大学);Tencent(腾讯);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Peking University(北京大学);Lancaster University(兰卡斯特大学)
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院);University of Electronic Science and Technology of China(电子科技大学);Wuhan Vocational College of Software and Engineering(武汉软件工程职业学院)
CoReS: Orchestrating the Dance of Reasoning and Segmentation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Luoyang Institute for Robot and Intelligent Equipment(洛阳机器人与智能装备研究所);Alibaba Group(阿里巴巴集团);Northeastern University(东北大学);Ant Group(蚂蚁集团)
PSGR: Pixel-wise Sparse Graph Reasoning for COVID-19 Pneumonia Segmentation in CT Images
Northwestern Polytechnical University(西北工业大学);Intel Labs(英特尔实验室);The University of Sydney(悉尼大学);University of Pittsburgh(匹兹堡大学);JD Finance America Corporation(京东金融美国公司)
Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced(深圳大学计算机科学与人工智能学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Electrical and Information Engineering, Northeast Petroleum University(东北石油大学电气与信息工程学院)
School of Electronic Information, Wuhan University of Science and Technology(武汉科技大学电子信息学院);State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(北京邮电大学网络与交换技术国家重点实验室);Oriental Space Port Research Institute(东方航天港研究院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
School of Automation, Southeast University, Nanjing, China;Baidu Inc, Beijing, China;College of Information Science \& Electronic Engineering, Zhejiang University, Hangzhou, China
ArgusCogito: Chain-of-Thought for Cross-Modal Synergy and Omnidirectional Reasoning in Camouflaged Object Segmentation
University of Chinese Academy of Sciences(中国科学院大学);Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间利用技术与工程中心)