像素对齐语言模型
Pixel Aligned Language Models
AI总结：
针对大语言模型难以处理视觉定位任务的问题，提出像素对齐语言模型，支持位置输入输出，在多位置感知视觉任务上取得SOTA性能。
摘要：
近年来大语言模型取得了巨大成功，其视觉领域的变体同样成果显著。现有视觉-语言模型能够用自然语言描述图像、回答视觉相关问题或对图像进行复杂推理，但目前尚不清楚如何利用大语言模型完成词接地、指代定位等定位任务。本研究旨在开发一种可将点集、边界框等位置信息作为输入或输出的视觉-语言模型：当位置作为输入时，模型执行位置条件字幕生成，为指定对象或区域生成字幕；当位置作为输出时，模型为语言模型生成的每个输出词回归像素坐标，从而实现密集词接地。该模型在Localized Narrative（局部叙事）数据集上预训练，该数据集包含基于人类注意力的像素-词对齐字幕。实验表明，模型可应用于指代定位、位置条件字幕生成、密集对象字幕生成等多种位置感知视觉-语言任务，在RefCOCO和Visual Genome基准上取得了当前最优性能。项目主页：https://jerryxu.net/PixelLLM 。
英文摘要：
Large language models have achieved great success in recent years, so as their variants in vision. Existing vision-language models can describe images in natural languages, answer visual-related questions, or perform complex reasoning about the image. However, it is yet unclear how localization tasks, such as word grounding or referring localization, can be performed using large language models. In this work, we aim to develop a vision-language model that can take locations, for example, a set of points or boxes, as either inputs or outputs. When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region. When generating locations as outputs, our model regresses pixel coordinates for each output word generated by the language model, and thus performs dense word grounding. Our model is pre-trained on the Localized Narrative dataset, which contains pixel-word-aligned captioning from human attention. We show our model can be applied to various location-aware vision-language tasks, including referring localization, location-conditioned captioning, and dense object captioning, archiving state-of-the-art performance on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
