arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.00692cs.CV
收藏

LISA:通过大语言模型进行推理分割

LISA: Reasoning Segmentation via Large Language Model

CUHK(香港中文大学);MSRA(微软亚洲研究院);SmartMore(旷世科技);HIT (Shenzhen)(哈尔滨工业大学(深圳))

Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, Jiaya Jia

AI总结:

本文提出推理分割任务及基准,并设计多模态大语言模型LISA,通过<SEG>标记和embedding-as-mask范式使LLM具备分割掩码生成能力,实现复杂隐含指令下的推理分割。

摘要:

尽管感知系统近年来取得了显著进展,它们仍然依赖明确的人类指令或预定义类别来在执行视觉识别任务之前识别目标对象。此类系统无法主动推理和理解隐含的用户意图。在这项工作中,我们提出了一个新的分割任务——推理分割(reasoning segmentation)。该任务旨在给定复杂且隐含的查询文本时输出分割掩码。此外,我们建立了一个包含超过一千个图像-指令-掩码数据样本的基准,其中融入了复杂的推理和世界知识以用于评估。最后,我们提出了LISA:大语言指令分割助手(large Language Instructed Segmentation Assistant),它继承了多模态大语言模型(LLMs)的语言生成能力,同时也具备生成分割掩码的能力。我们用一个<SEG>标记扩展了原始词汇表,并提出了embedding-as-mask范式以解锁分割能力。值得注意的是,LISA能够处理涉及复杂推理和世界知识的情况。同时,当仅在无推理数据集上训练时,它展现出稳健的零样本能力。此外,仅用239个推理分割数据样本对模型进行微调即可进一步提升性能。定量和定性实验均表明,我们的方法有效地为多模态LLMs解锁了新的推理分割能力。代码、模型和数据可在https://github.com/dvlab-research/LISA获取。

英文摘要:

Although perception systems have made remarkable advancements in recent years, they still rely on explicit human instruction or pre-defined categories to identify the target objects before executing visual recognition tasks. Such systems cannot actively reason and comprehend implicit user intention. In this work, we propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction-mask data samples, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of multimodal Large Language Models (LLMs) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a <SEG> token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving complex reasoning and world knowledge. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation data samples results in further performance enhancement. Both quantitative and qualitative experiments show our method effectively unlocks new reasoning segmentation capabilities for multimodal LLMs. Code, models, and data are available at https://github.com/dvlab-research/LISA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑