arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20177cs.CLcs.CV
收藏

从感知到思考:解耦感知与推理提升视觉语言模型的训练

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Amazon(亚马逊);University of Waterloo(滑铁卢大学);Vector Institute, Canada CIFAR AI Chair(向量研究所,加拿大CIFAR人工智能主席)

Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang, Freda Shi, Hui Liu, Hanqing Lu, Cihang Xie, Yuyin Zhou

AI总结:

本研究通过解耦感知与推理,发现视觉任务性能受限于感知能力不足而非推理本身,通过分阶段训练提升模型的感知与推理能力,从而在多个视觉数学和感知任务中取得更优表现。

摘要:

最近视觉语言模型(VLMs)的进步强调长链推理;然而,我们发现其在视觉任务上的性能主要受限于感知能力不足而非推理本身。在本工作中,我们系统研究了VLMs在训练后感知与推理之间的相互作用,通过将能力分解为三个独立的训练阶段:视觉感知、视觉推理和文本推理,并结合专门的训练数据。我们证明视觉感知(a)需要针对优化和专门数据;(b)作为基础框架,应在细化视觉推理之前通过分阶段训练巩固;(c)通过强化学习(RL)比基于标题的监督微调(SFT)更有效学习。我们的实验表明,分阶段训练在多个VLMs上一致提升了视觉感知和推理性能。值得注意的是,采用我们方法训练的模型在推理准确性上提高了1.5%,推理轨迹缩短了20.8%,表明更强的感知减少了对过度推理的需求。此外,我们展示了基于能力的分阶段训练代表了与传统难度基于课程正交的新课程维度,结合两者可进一步获得加性收益。我们的分阶段训练模型在开放权重VLMs中表现优异,在多个视觉数学和感知任务(如WeMath和RealWorldQA)上取得了优于基础模型的先进结果。

英文摘要:

Recent advances in vision-language models (VLMs) emphasize long chain-of-thought reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to reasoning itself. In this work, we systematically study the interplay between perception and reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual reasoning, and textual reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces, suggesting that superior perception reduces the need for excessive reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑