arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.21037cs.LGcs.AIcs.CLcs.CV
收藏

基于框架的思考:视觉上下文和测试时扩展如何增强视频推理

Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning

University of Cambridge(剑桥大学);Pioneer Center for AI, University of Copenhagen(人工智能先锋中心,哥本哈根大学);Hong Kong University of Science(香港科学大学);University of California San Diego(加州大学圣地亚哥分校);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Peking University(北京大学)

Chengzu Li, Zanyi Wang, Jiaang Li, Yi Xu, Han Zhou, Huanyu Zhang, Ruichuan An, Dengyang Jiang, Zhaochong An, Ivan Vulić, Serge Belongie, Anna Korhonen

AI总结:

本文提出通过视频生成模型进行视觉推理,利用视觉上下文和测试时扩展提升视频推理能力,展示了模型在空间和时间复杂任务中的鲁棒零样本泛化能力。

摘要:

视觉-语言模型在文本推理方面表现出色,但它们在细粒度空间理解和连续动作规划方面常常遇到困难,无法模拟复杂视觉推理所需的动力学。在本工作中,我们通过视频生成模型来制定视觉推理,认为生成的帧可以作为初始状态和解决方案之间的中间推理步骤。我们评估了其在两个不同的领域中的能力:用于具有低视觉变化的顺序离散规划的迷宫导航,以及用于具有高视觉变化的连续操作的乐高拼图。我们的实验揭示了三个关键发现:(1)鲁棒零样本泛化:在两项任务中,模型在未见过的数据分布上表现出色,无需特定微调。(2)视觉上下文:模型有效利用视觉上下文作为显式控制,例如代理图标和乐高形状,使其能够保持高水平的视觉一致性和适应未见过的模式。(3)视觉测试时扩展:我们在顺序规划中观察到测试时扩展定律;增加生成视频的长度(视觉推理预算)能增强对空间和时间复杂路径的零样本泛化能力。这些发现表明,视频生成不仅仅是媒体工具,而是一种可扩展、可泛化的视觉推理范式。

英文摘要:

Vision-Language Models have excelled at textual reasoning, but they often struggle with fine-grained spatial understanding and continuous action planning, failing to simulate the dynamics required for complex visual reasoning. In this work, we formulate visual reasoning by means of video generation models, positing that generated frames can act as intermediate reasoning steps between initial states and solutions. We evaluate their capacity in two distinct regimes: Maze Navigation for sequential discrete planning with low visual change and Tangram Puzzle for continuous manipulation with high visual change. Our experiments reveal three critical insights: (1) Robust Zero-Shot Generalization: In both tasks, the model demonstrates strong performance on unseen data distributions without specific finetuning. (2) Visual Context: The model effectively uses visual context as explicit control, such as agent icons and tangram shapes, enabling it to maintain high visual consistency and adapt its planning capability robustly to unseen patterns. (3) Visual Test-Time Scaling: We observe a test-time scaling law in sequential planning; increasing the generated video length (visual inference budget) empowers better zero-shot generalization to spatially and temporally complex paths. These findings suggest that video generation is not merely a media tool, but a scalable, generalizable paradigm for visual reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑