缓解大多模态模型中的视觉上下文退化：一种无需训练的解耦代理框架
Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework
AI总结：
本文提出无需训练的解耦代理框架DRP，通过让LLM作为策略推理者与LMM作为观察者解耦，有效缓解多模态推理中的视觉退化问题，实验表明其在MathVision基准上准确率优于GPT-4o。
摘要：
随着大语言模型（LLMs）的持续扩展和强化学习的进步，LLMs已展现出卓越的推理能力，能够解决广泛复杂的难题。受这些成就的启发，研究者将相关技术扩展到了大多模态模型（LMMs）。然而，一个关键的限制出现了，体现在视觉基础的逐步丧失。随着推理链的增长，LMMs倾向于越来越多地依赖早期步骤生成的文本信息，而最初提取的视觉信息很少被重新访问或整合。这种现象常导致推理过程偏离实际图像内容，产生视觉不合理的甚至错误的结论。为克服这一根本限制，我们提出了一种新颖的、无需训练的代理范式，即解耦认知推理与视觉感知（DRP）。在此框架中，一个强大的LLM作为策略推理者，通过显式查询LMM（作为专门的观察者）来获取所需的细粒度视觉细节。该方法轻量、模型无关且即插即用，不需要额外的训练或架构修改。广泛的实验表明，我们的框架DRP在调节视觉推理轨迹、显著缓解推理漂移和强制稳健视觉基础方面表现出色。值得注意的是，在MathVision基准上，Qwen2.5-VL-7B和Qwen3-32B的结合实现了47.2%的准确率，优于GPT-4o的40.6%。这些发现凸显了我们的方法在无需昂贵重新训练的情况下提升多模态推理可靠性的潜力。我们的代码可在https://github.com/hongruijia/DRP公开获取。
英文摘要：
With the continuous expansion of Large Language Models (LLMs) and advances in reinforcement learning, LLMs have demonstrated exceptional reasoning capabilities, enabling them to address a wide range of complex problems. Inspired by these achievements, researchers have extended related techniques to Large Multimodal Models (LMMs). However, a critical limitation has emerged, reflected in the progressive loss of visual grounding. As the reasoning chain grows longer, LMMs tend to rely increasingly on the textual information generated in earlier steps, while the initially extracted visual information is rarely revisited or incorporated. This phenomenon often causes the reasoning process to drift away from the actual image content, resulting in visually implausible or even erroneous conclusions. To overcome this fundamental limitation, we propose a novel, training-free agentic paradigm that Decouples cognitive Reasoning from visual Perception (DRP). In this framework, a powerful LLM serves as a strategic Reasoner, orchestrating the inference process by explicitly querying an LMM-acting as a dedicated Observer-to retrieve fine-grained visual details on demand. This approach is lightweight, model-agnostic, and plug-and-play, necessitating no additional training or architectural modifications. Extensive experiments demonstrate our framework DRP's efficacy in regulating the visual reasoning trajectory, significantly mitigating reasoning drift, and enforcing robust visual grounding. Notably, on the MathVision benchmark, the integration of Qwen2.5-VL-7B and Qwen3-32B achieves an accuracy of 47.2\%, outperforming GPT-4o's 40.6\%. These findings underscore the potential of our approach to enhance multimodal reasoning reliability without the need for costly retraining. Our code is publicly available at https://github.com/hongruijia/DRP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
