CMMCoT：通过多模态链式思考和记忆增强提升复杂多图像理解
CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation
AI总结：
CMMCoT通过多模态链式思考和记忆增强提升多图像理解的复杂性与准确性
摘要：
尽管先前的多模态慢思考方法在单图像理解场景中表现出色，但当扩展到更复杂的多图像理解任务时，其效果受到根本限制。这种限制源于其主要依赖于基于文本的中间推理过程。对于人类而言，在进行复杂的多图像分析时，通常会执行两种互补的认知操作：（1）通过感兴趣区域匹配进行连续的跨图像视觉比较；（2）在推理链中动态记忆关键视觉概念。受这些观察的启发，我们提出了复杂的多模态链式思考（CMMCoT）框架，这是一个多步骤推理框架，旨在模仿人类般的“慢思考”以进行多图像理解。我们的方法引入了两个关键创新：（1）构建交错的多模态多步骤推理链，利用从中间推理步骤中提取的关键视觉区域标记作为监督信号。这种机制不仅促进了全面的跨模态理解，还增强了模型的可解释性。（2）引入了一个测试时的记忆增强模块，该模块在推理过程中扩展了模型的推理能力，同时保持参数效率。此外，为了促进该方向的研究，我们还整理了一个新的多图像慢思考数据集。广泛的实验证明了我们模型的有效性。代码可在https://github.com/zhangguanghao523/CMMCoT上获得。
英文摘要：
While previous multimodal slow-thinking methods have demonstrated remarkable success in single-image understanding scenarios, their effectiveness becomes fundamentally constrained when extended to more complex multi-image comprehension tasks. This limitation stems from their predominant reliance on text-based intermediate reasoning processes. While for human, when engaging in sophisticated multi-image analysis, they typically perform two complementary cognitive operations: (1) continuous cross-image visual comparison through region-of-interest matching, and (2) dynamic memorization of critical visual concepts throughout the reasoning chain. Motivated by these observations, we propose the Complex Multi-Modal Chain-of-Thought (CMMCoT) framework, a multi-step reasoning framework that mimics human-like "slow thinking" for multi-image understanding. Our approach incorporates two key innovations: (1) The construction of interleaved multimodal multi-step reasoning chains, which utilize critical visual region tokens, extracted from intermediate reasoning steps, as supervisory signals. This mechanism not only facilitates comprehensive cross-modal understanding but also enhances model interpretability. (2) The introduction of a test-time memory augmentation module that expands the model's reasoning capacity during inference while preserving parameter efficiency. Furthermore, to facilitate research in this direction, we have curated a novel multi-image slow-thinking dataset. Extensive experiments demonstrate the effectiveness of our model. Code is available at https://github.com/zhangguanghao523/CMMCoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
