ContextualLVLM-Agent：面向多轮视觉对话与复杂指令遵循的整体框架
ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following
AI总结：
针对现有多模态模型在多轮视觉对话中上下文丢失和推理不足的问题，本文提出MMDR-Bench基准和CoLVLM Agent框架，通过记忆-感知-规划-执行迭代循环增强LVLMs，无需重训练即在人类评估中超越GPT-4o和Gemini 1.5 Pro。
摘要：
尽管大语言模型（LLMs）和大视觉语言模型（LVLMs）取得了显著进展，但当前模型在处理复杂的、多轮的、视觉锚定的任务时仍面临重大挑战，这些任务要求深度推理、持续的上下文理解、实体跟踪以及多步指令遵循。现有基准往往难以捕捉真实世界多模态交互的动态性和复杂性，从而导致上下文丢失和视觉幻觉等问题。为解决这些局限，我们提出了MMDR-Bench（多模态对话推理基准），这是一个新颖的数据集，包含300个精心设计的复杂多轮对话场景，每个场景平均5-7轮，并围绕视觉实体跟踪和推理深度等六个核心维度进行评估。此外，我们提出了CoLVLM Agent（上下文LVLM智能体），这是一个整体框架，通过迭代的“记忆-感知-规划-执行”循环增强现有LVLMs的高级推理和指令遵循能力，且无需对底层模型进行大规模重新训练。我们在MMDR-Bench上的大量实验表明，CoLVLM Agent持续取得优越性能，平均人类评估得分达到4.03，显著超越GPT-4o（3.92）和Gemini 1.5 Pro（3.85）等最先进的商业模型。该框架在推理深度、指令遵循和错误抑制方面表现出显著优势，并在延长对话轮次中保持稳健性能，验证了其模块化设计和迭代方法在复杂多模态交互中的有效性。
英文摘要：
Despite significant advancements in Large Language Models (LLMs) and Large Vision-Language Models (LVLMs), current models still face substantial challenges in handling complex, multi-turn, and visually-grounded tasks that demand deep reasoning, sustained contextual understanding, entity tracking, and multi-step instruction following. Existing benchmarks often fall short in capturing the dynamism and intricacies of real-world multi-modal interactions, leading to issues such as context loss and visual hallucinations. To address these limitations, we introduce MMDR-Bench (Multi-Modal Dialogue Reasoning Benchmark), a novel dataset comprising 300 meticulously designed complex multi-turn dialogue scenarios, each averaging 5-7 turns and evaluated across six core dimensions including visual entity tracking and reasoning depth. Furthermore, we propose CoLVLM Agent (Contextual LVLM Agent), a holistic framework that enhances existing LVLMs with advanced reasoning and instruction following capabilities through an iterative "memory-perception-planning-execution" cycle, requiring no extensive re-training of the underlying models. Our extensive experiments on MMDR-Bench demonstrate that CoLVLM Agent consistently achieves superior performance, attaining an average human evaluation score of 4.03, notably surpassing state-of-the-art commercial models like GPT-4o (3.92) and Gemini 1.5 Pro (3.85). The framework exhibits significant advantages in reasoning depth, instruction adherence, and error suppression, and maintains robust performance over extended dialogue turns, validating the effectiveness of its modular design and iterative approach for complex multi-modal interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
