基于多模态大语言模型的上下文目标检测
Contextual Object Detection with Multimodal Large Language Models
AI总结：
本文提出ContextDET模型，通过端到端可微的视觉-语言上下文建模，解决多模态大语言模型缺乏目标检测能力的问题，并在多个基准上验证其有效性。
摘要：
近年来，多模态大语言模型（MLLMs）在图像描述和视觉问答等视觉-语言任务中表现卓越，但缺乏基本的感知能力，即目标检测。本文通过引入一个新的研究问题——上下文目标检测，来弥补这一不足，即在不同的人机交互上下文中理解可见目标。我们研究了三种代表性场景，包括语言完形填空、视觉描述和问答。此外，我们提出了ContextDET，这是一个统一的多模态模型，能够对视觉-语言上下文进行端到端的可微建模，从而在人机交互中定位、识别视觉目标并将其与语言输入相关联。我们的ContextDET包含三个关键子模型：（i）用于提取视觉表示的视觉编码器，（ii）用于多模态上下文解码的预训练LLM，以及（iii）用于根据上下文目标词预测边界框的视觉解码器。新的“先生成后检测”框架使我们能够检测人类词汇表中的目标词。大量实验表明，ContextDET在我们提出的CODE基准、开放词汇检测和指代表达图像分割上具有优势。Github: https://github.com/yuhangzang/ContextDET。
英文摘要：
Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, we address this limitation by introducing a novel research problem of contextual object detection -- understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering. Moreover, we present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts, so as to locate, identify, and associate visual objects with language inputs for human-AI interaction. Our ContextDET involves three key submodels: (i) a visual encoder for extracting visual representations, (ii) a pre-trained LLM for multimodal context decoding, and (iii) a visual decoder for predicting bounding boxes given contextual object words. The new generate-then-detect framework enables us to detect object words within human vocabulary. Extensive experiments show the advantages of ContextDET on our proposed CODE benchmark, open-vocabulary detection, and referring image segmentation. Github: https://github.com/yuhangzang/ContextDET.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
