arXiv 2505.155102025-10-26cs.CVcs.CL

视觉思维：理解多模态思维链的统一视角

Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought

School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology（哈尔滨工业大学社会计算与交互机器人研究中心）；Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳研究院计算与智能研究所）；Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center, Guizhou University（贵州大学文字计算与认知智能教育部工程研究中心）；Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；ByteDance Seed (China)（字节跳动种子（中国））

Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, Libo Qin

备注：: Accepted at NeurIPS 2025;

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出统一视角理解多模态思维链，揭示其通过视觉思维增强性能，定义四种表达形式并分析其影响，为未来研究提供启示。

摘要：

大型视觉语言模型（LVLMs）在多模态任务中取得了显著成功，多模态思维链（MCoT）进一步提升了性能与可解释性。近期MCoT方法分为两类：（i）文本型MCoT（T-MCoT），接收多模态输入并产生文本输出；（ii）交错型MCoT（I-MCoT），生成交错图像-文本输出。尽管两种方法均有进展，但其性能提升的机制尚未被完全理解。为填补这一空白，我们首先揭示MCoT通过融入视觉思维来增强LVLMs，视觉思维将图像信息传递给推理过程，且与MCoT格式无关，仅依赖于表达的清晰性与简洁性。此外，为系统探索视觉思维，我们定义了四种不同的视觉思维表达形式并进行了全面分析。我们的发现表明，这些形式在清晰性与简洁性上存在差异，从而带来不同程度的MCoT改进。进一步，我们探究了视觉思维的内部本质，发现视觉思维作为输入图像与深层Transformer层推理之间的中介，实现更高级的视觉信息传递。我们希望视觉思维能激发未来MCoT研究的进一步突破。

英文摘要：

Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。