arXiv 2411.102522024-11-15cs.CV

视觉-语言智能体：迈向协作式上下文对象推理

Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning

Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；Beijing Institute of Petrochemical Technology（北京石油化工学院）；TikTok（抖音）；Sealand Technology Inc.（Sealand科技公司）；Singapore Management University（新加坡管理大学）

Jingru Yang, Huan Yu, Yang Jingxin, Chentianye Xu, Yin Biao, Yu Sun, Shengfeng He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出视觉-语言智能体（VLA）协作框架，结合MLLM的关系推理与传统检测器的定位能力，通过语言智能体与视觉智能体协同优化检测结果，在COCO数据集上显著提升了多模型的物体定位与上下文连贯性。

摘要：

多模态大语言模型（MLLMs）擅长图像内的描述性任务，但通常难以实现精确的物体定位，而这正是可靠视觉解释的关键要素。相比之下，传统的物体检测模型提供了高定位精度，但由于对物体间关系的建模有限，经常产生缺乏上下文连贯性的检测结果。为了解决这一根本局限，我们引入了视觉-语言智能体（VLA），这是一个结合了MLLMs关系推理优势与传统物体检测器精确定位能力的协作框架。在VLA范式中，MLLM作为核心的语言智能体，与专门用于物体检测和分类的视觉智能体协同工作。语言智能体通过对物体间的空间和上下文关系进行推理来评估并优化检测结果，而分类视觉智能体则提供纠正性反馈以提高分类准确度。这种协作方法使VLA能够显著增强空间推理和物体定位能力，解决多模态理解中的关键挑战。在COCO数据集上的广泛评估表明，多个检测模型均取得了显著的性能提升，凸显了VLA在准确且上下文连贯的物体检测方面树立新基准的潜力。

英文摘要：

Multimodal Large Language Models (MLLMs) excel at descriptive tasks within images but often struggle with precise object localization, a critical element for reliable visual interpretation. In contrast, traditional object detection models provide high localization accuracy but frequently generate detections lacking contextual coherence due to limited modeling of inter-object relationships. To address this fundamental limitation, we introduce the \textbf{Visual-Linguistic Agent (VLA), a collaborative framework that combines the relational reasoning strengths of MLLMs with the precise localization capabilities of traditional object detectors. In the VLA paradigm, the MLLM serves as a central Linguistic Agent, working collaboratively with specialized Vision Agents for object detection and classification. The Linguistic Agent evaluates and refines detections by reasoning over spatial and contextual relationships among objects, while the classification Vision Agent offers corrective feedback to improve classification accuracy. This collaborative approach enables VLA to significantly enhance both spatial reasoning and object localization, addressing key challenges in multimodal understanding. Extensive evaluations on the COCO dataset demonstrate substantial performance improvements across multiple detection models, highlighting VLA's potential to set a new benchmark in accurate and contextually coherent object detection.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。