NExT-Chat：一种用于对话、检测与分割的大多模态模型（LMM）
NExT-Chat: An LMM for Chat, Detection and Segmentation
AI总结：
本研究提出pix2emb新型物体位置建模范式，训练出支持多位置格式的多模态模型NExT-Chat，在视觉定位、分割、区域描述等多项任务上性能优于现有模型，并开源了相关资源。
摘要：
大语言模型（LLM）的发展极大推动了多模态理解领域的进步，催生了大多模态模型（LMMs）。为提升视觉理解水平，近期研究通过将物体边界框坐标表示为一系列文本序列（pix2seq），赋予LMMs区域级理解能力。本文提出一种名为pix2emb方法的新型物体位置建模范式，让LMM输出位置嵌入，再用不同解码器对其进行解码。该范式支持在多模态对话中使用不同位置格式（如边界框和掩码）。依托提出的pix2emb方法，我们训练了名为NExT-Chat的LMM，并验证了其处理视觉定位、区域描述、定位推理等多项任务的能力。综合实验证明了NExT-Chat在各类任务上的有效性，例如：在POPE-Random基准上，NExT-Chat（87.7）优于Shikra（86.9）；在指代表达分割任务上，NExT-Chat（68.9）优于LISA（67.9）；在区域描述任务上，NExT-Chat（79.6）优于Kosmos-2（62.3）。代码和模型已发布于https://github.com/NExT-ChatV/NExT-Chat。
英文摘要：
The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
