LMM-Det：让大型多模态模型在目标检测中表现卓越
LMM-Det: Make Large Multimodal Models Excel in Object Detection
AI总结：
针对LMM目标检测能力弱于专用检测器的问题，提出无需专用检测模块的LMM-Det，通过数据分布调整、推理优化和指令对话重组提升召回率与检测性能。
摘要：
大型多模态模型（LMMs）凭借其在多模态理解、推理和上下文学习等方面的卓越能力，在人工智能研究界和工业界获得了广泛关注与兴趣。尽管LMMs在处理图像描述、视觉问答和视觉定位等多模态任务时已展现出令人期待的结果，但与专用检测器相比，LMMs的目标检测能力仍存在显著差距。为弥合这一差距，我们摒弃了将重型检测器与LMMs集成的传统方法，提出了LMM-Det，这是一种简单而有效的方法，利用大型多模态模型进行普通目标检测，而无需依赖专门的检测模块。具体而言，我们对大型多模态模型面对目标检测时的情况进行了全面的探索性分析，揭示出其召回率相比专用检测模型显著下降。为缓解这一问题，我们提出通过引入面向目标检测定制的数据分布调整和推理优化来提高召回率。我们重新组织指令对话，以增强大型多模态模型的目标检测能力。我们主张，大型多模态模型在没有任何额外检测模块的情况下也具备检测能力。大量实验支持了我们的主张，并展示了通用LMM-Det的有效性。数据集、模型和代码可在https://github.com/360CVGroup/LMM-Det获取。
英文摘要：
Large multimodal models (LMMs) have garnered wide-spread attention and interest within the artificial intelligence research and industrial communities, owing to their remarkable capability in multimodal understanding, reasoning, and in-context learning, among others. While LMMs have demonstrated promising results in tackling multimodal tasks like image captioning, visual question answering, and visual grounding, the object detection capabilities of LMMs exhibit a significant gap compared to specialist detectors. To bridge the gap, we depart from the conventional methods of integrating heavy detectors with LMMs and propose LMM-Det, a simple yet effective approach that leverages a Large Multimodal Model for vanilla object Detection without relying on specialized detection modules. Specifically, we conduct a comprehensive exploratory analysis when a large multimodal model meets with object detection, revealing that the recall rate degrades significantly compared with specialist detection models. To mitigate this, we propose to increase the recall rate by introducing data distribution adjustment and inference optimization tailored for object detection. We re-organize the instruction conversations to enhance the object detection capabilities of large multimodal models. We claim that a large multimodal model possesses detection capability without any extra detection modules. Extensive experiments support our claim and show the effectiveness of the versatile LMM-Det. The datasets, models, and codes are available at https://github.com/360CVGroup/LMM-Det.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
