arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.22010cs.CV
收藏

CoFFT:面向视觉语言模型的前瞻-聚焦思维链

CoFFT: Chain of Foresight-Focus Thought for Visual Language Models

School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China(教育部智能网络与网络安全重点实验室);Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China(陕西省大数据知识工程重点实验室);IHPC, Agency for Science, Technology and Research, Singapore(新加坡科技研究局IHPC);Show Lab, National University of Singapore(新加坡国立大学Show实验室);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院)

Xinyu Zhang, Yuxuan Dong, Lingling Zhang, Chengyou Jia, Zhuohang Dang, Basura Fernando, Jun Liu, Mike Zheng Shou

AI总结:

针对视觉语言模型易受图像冗余信息干扰的问题,提出无需训练的前瞻-聚焦思维链CoFFT,通过三阶段迭代循环模拟人类视觉认知,在多个基准上实现3.1%-5.8%的性能提升。

摘要:

尽管视觉语言模型(Vision Language Models, VLMs)已取得显著进展,但它们仍受限于视觉输入的复杂性与冗余性。当图像包含大量无关信息时,VLM 易受干扰,从而产生过多与任务无关的推理过程,甚至出现幻觉。这一局限源于它们无法在推理过程中精准发现并处理所需区域。为解决这一问题,我们提出前瞻-聚焦思维链(Chain of Foresight-Focus Thought, CoFFT),这是一种无需训练的新方法,通过模拟人类视觉认知增强 VLM 的视觉推理能力。每个前瞻-聚焦思维单元包含三个阶段:(1)多样样本生成:生成多样化的推理样本以探索潜在推理路径,每个样本包含若干推理步骤;(2)双前瞻解码:基于视觉聚焦度和推理进展对这些样本进行严格评估,将最优样本的第一步加入推理过程;(3)视觉聚焦调整:将视觉聚焦精准调整至对未来推理最有利的区域,随后返回阶段(1)生成后续推理样本,直至得到最终答案。这些阶段迭代运行,形成一个相互依赖的循环:推理引导视觉聚焦,视觉聚焦又为后续推理提供依据。在 Qwen2.5-VL、InternVL-2.5 和 Llava-Next 上的多个基准测试实验结果表明,该方法在计算开销增加可控的情况下,性能持续提升 3.1%-5.8%。

英文摘要:

Despite significant advances in Vision Language Models (VLMs), they remain constrained by the complexity and redundancy of visual input. When images contain large amounts of irrelevant information, VLMs are susceptible to interference, thus generating excessive task-irrelevant reasoning processes or even hallucinations. This limitation stems from their inability to discover and process the required regions during reasoning precisely. To address this limitation, we present the Chain of Foresight-Focus Thought (CoFFT), a novel training-free approach that enhances VLMs' visual reasoning by emulating human visual cognition. Each Foresight-Focus Thought consists of three stages: (1) Diverse Sample Generation: generates diverse reasoning samples to explore potential reasoning paths, where each sample contains several reasoning steps; (2) Dual Foresight Decoding: rigorously evaluates these samples based on both visual focus and reasoning progression, adding the first step of optimal sample to the reasoning process; (3) Visual Focus Adjustment: precisely adjust visual focus toward regions most beneficial for future reasoning, before returning to stage (1) to generate subsequent reasoning samples until reaching the final answer. These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning. Empirical results across multiple benchmarks using Qwen2.5-VL, InternVL-2.5, and Llava-Next demonstrate consistent performance improvements of 3.1-5.8% with controllable increasing computational overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑