CoFFT:面向视觉语言模型的前瞻-聚焦思维链
CoFFT: Chain of Foresight-Focus Thought for Visual Language Models
AI总结:
针对视觉语言模型易受图像冗余信息干扰的问题,提出无需训练的前瞻-聚焦思维链CoFFT,通过三阶段迭代循环模拟人类视觉认知,在多个基准上实现3.1%-5.8%的性能提升。
摘要:
尽管视觉语言模型(Vision Language Models, VLMs)已取得显著进展,但它们仍受限于视觉输入的复杂性与冗余性。当图像包含大量无关信息时,VLM 易受干扰,从而产生过多与任务无关的推理过程,甚至出现幻觉。这一局限源于它们无法在推理过程中精准发现并处理所需区域。为解决这一问题,我们提出前瞻-聚焦思维链(Chain of Foresight-Focus Thought, CoFFT),这是一种无需训练的新方法,通过模拟人类视觉认知增强 VLM 的视觉推理能力。每个前瞻-聚焦思维单元包含三个阶段:(1)多样样本生成:生成多样化的推理样本以探索潜在推理路径,每个样本包含若干推理步骤;(2)双前瞻解码:基于视觉聚焦度和推理进展对这些样本进行严格评估,将最优样本的第一步加入推理过程;(3)视觉聚焦调整:将视觉聚焦精准调整至对未来推理最有利的区域,随后返回阶段(1)生成后续推理样本,直至得到最终答案。这些阶段迭代运行,形成一个相互依赖的循环:推理引导视觉聚焦,视觉聚焦又为后续推理提供依据。在 Qwen2.5-VL、InternVL-2.5 和 Llava-Next 上的多个基准测试实验结果表明,该方法在计算开销增加可控的情况下,性能持续提升 3.1%-5.8%。
英文摘要:
Despite significant advances in Vision Language Models (VLMs), they remain constrained by the complexity and redundancy of visual input. When images contain large amounts of irrelevant information, VLMs are susceptible to interference, thus generating excessive task-irrelevant reasoning processes or even hallucinations. This limitation stems from their inability to discover and process the required regions during reasoning precisely. To address this limitation, we present the Chain of Foresight-Focus Thought (CoFFT), a novel training-free approach that enhances VLMs' visual reasoning by emulating human visual cognition. Each Foresight-Focus Thought consists of three stages: (1) Diverse Sample Generation: generates diverse reasoning samples to explore potential reasoning paths, where each sample contains several reasoning steps; (2) Dual Foresight Decoding: rigorously evaluates these samples based on both visual focus and reasoning progression, adding the first step of optimal sample to the reasoning process; (3) Visual Focus Adjustment: precisely adjust visual focus toward regions most beneficial for future reasoning, before returning to stage (1) to generate subsequent reasoning samples until reaching the final answer. These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning. Empirical results across multiple benchmarks using Qwen2.5-VL, InternVL-2.5, and Llava-Next demonstrate consistent performance improvements of 3.1-5.8% with controllable increasing computational overhead.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。