arXiv 2406.193892024-10-01cs.CV

OMG-LLaVA：桥接图像级、对象级、像素级推理与理解

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Wuhan University（武汉大学）；Skywork AI（天际工作人工智能）；Bytedance（字节跳动）

Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, Shuicheng Yan

备注：: NeurIPS-2024. Project page: https://lxtgh.github.io/project/omg_llava/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出OMG-LLaVA框架，将通用分割编码器与LLM结合，通过感知先验嵌入在单一模型中统一实现图像级、对象级和像素级推理与理解，并在多个基准上达到或超越专用方法。

摘要：

当前的通用分割方法在像素级图像和视频理解方面展现出强大能力，但它们缺乏推理能力，且无法通过文本指令进行控制。相比之下，大型视觉-语言多模态模型具有强大的基于视觉的对话和推理能力，但缺乏像素级理解，并且难以接受视觉提示以实现灵活的用户交互。本文提出OMG-LLaVA，一个新颖而优雅的框架，将强大的像素级视觉理解与推理能力相结合。它可以接受各种视觉和文本提示，实现灵活的用户交互。具体而言，我们使用通用分割方法作为视觉编码器，将图像信息、感知先验和视觉提示整合为提供给LLM的视觉token。LLM负责理解用户的文本指令，并根据视觉信息提供文本响应和像素级分割结果。我们提出感知先验嵌入，以更好地将感知先验与图像特征相融合。OMG-LLaVA在单一模型中实现了图像级、对象级和像素级的推理与理解，在多个基准测试上达到或超越了专用方法的性能。我们的工作不是使用LLM来连接各个专家模型，而是致力于在一个编码器、一个解码器和一个LLM上进行端到端训练。代码和模型已发布以供进一步研究。

英文摘要：

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。