arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23959cs.CV
收藏

通过图像作为连续动作进行思考:数值视觉链式推理

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);Chalmers University of Technology(楚克理工大学);University of Gothenburg(哥德堡大学)

Kesen Zhao, Beier Zhu, Junbao Zhou, Xingyu Zhu, Zhongqi Yue, Hanwang Zhang

AI总结:

NV-CoT通过将图像推理动作空间扩展为连续欧几里得空间,提升MLLMs的定位精度和回答准确性,同时加速训练收敛。

摘要:

最近的多模态大语言模型(MLLMs)越来越多地依赖视觉链式推理来在图像上执行基于区域的推理。然而,现有方法通过文本化坐标来定位区域,导致模态不匹配和语义碎片化,或者使用固定粒度的块,这不仅限制了精确区域选择,而且通常需要非微不足道的架构更改。在本文中,我们提出了数值视觉链式推理(NV-CoT),一种框架,使MLLMs能够使用连续数值坐标进行图像推理。NV-CoT将MLLM的动作空间从离散词汇标记扩展到连续欧几里得空间,允许模型仅通过最小的架构修改直接生成边界框坐标作为动作。该框架支持监督微调和强化学习。特别是,我们用坐标上的高斯(或拉普拉斯)策略替代了分类标记策略,并通过重新参数化采样引入随机性,使NV-CoT完全兼容GRPO风格的策略优化。在三个基准测试上对八种代表性的视觉推理基线进行广泛实验,证明NV-CoT显著提高了定位精度和最终答案的准确性,同时加快了训练收敛速度,验证了连续动作视觉推理在MLLMs中的有效性。代码可在https://github.com/kesenzhao/NV-CoT上获得。

英文摘要:

Recent MLLMs increasingly rely on visual chain-of-thought to perform region-grounded reasoning over images. However, existing approaches ground regions via either textified coordinates (causing modality mismatch and semantic fragmentation) or fixed-granularity patches that both limit precise region selection and often require non-trivial architectural changes. In this paper, we propose Numerical Visual Chain-of-Thought (NV-CoT), a framework that enables MLLMs to reason over images using continuous numerical coordinates. NV-CoT expands the MLLM action space from discrete vocabulary tokens to a continuous Euclidean space, allowing models to directly generate bounding-box coordinates as actions with only minimal architectural modification. The framework supports both supervised fine-tuning and reinforcement learning. In particular, we replace categorical token policies with a Gaussian (or Laplace) policy over coordinates and introduce stochasticity via reparameterized sampling, making NV-CoT fully compatible with GRPO-style policy optimization. Extensive experiments on three benchmarks against 11 representative visual reasoning baselines demonstrate that NV-CoT significantly improves localization precision and final answer accuracy, while also accelerating training convergence, validating the effectiveness of continuous-action visual reasoning in MLLMs. Project page: https://kesenzhao.github.io/NV-CoT/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑