MathCanvas：多模态数学推理的内在视觉思维链
MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning
AI总结：
针对多模态数学推理中视觉思维链的不足，提出MathCanvas框架，通过两阶段训练赋予LMMs内在VCoT能力，在基准上相对提升86%。
摘要：
尽管大型语言模型（LLMs）在文本推理方面表现出色，但在几何等本质上依赖视觉辅助的数学领域却面临困难。现有的视觉思维链（VCoT）方法常常受限于僵化的外部工具，或无法生成解决复杂问题所需的高保真、策略性时机的图表。为弥补这一差距，我们引入了MathCanvas，一个旨在赋予统一大型多模态模型（LMMs）内在VCoT数学能力的综合框架。我们的方法包括两个阶段。首先，视觉操作阶段在新型的1520万对语料库上预训练模型，该语料库包含1000万对图文描述到图表对（MathCanvas-Imagen）和520万对逐步编辑轨迹（MathCanvas-Edit），以掌握图表生成和编辑。其次，策略性视觉辅助推理阶段在MathCanvas-Instruct上微调模型，这是一个包含21.9万个交错视觉-文本推理路径示例的新数据集，教导模型何时以及如何利用视觉辅助。为促进严格评估，我们引入了MathCanvas-Bench，一个包含3000个需要模型生成交错视觉-文本解决方案的挑战性基准。我们的模型BAGEL-Canvas在该框架下训练，在MathCanvas-Bench上相比强LMM基线实现了86%的相对改进，展示了在其他公开数学基准上的出色泛化能力。我们的工作提供了完整的工具包——框架、数据集和基准——以解锁LMMs中复杂的、类人的视觉辅助推理。项目页面：https://mathcanvas.github.io/
英文摘要：
While Large Language Models (LLMs) have excelled in textual reasoning, they struggle with mathematical domains like geometry that intrinsically rely on visual aids. Existing approaches to Visual Chain-of-Thought (VCoT) are often limited by rigid external tools or fail to generate the high-fidelity, strategically-timed diagrams necessary for complex problem-solving. To bridge this gap, we introduce MathCanvas, a comprehensive framework designed to endow unified Large Multimodal Models (LMMs) with intrinsic VCoT capabilities for mathematics. Our approach consists of two phases. First, a Visual Manipulation stage pre-trains the model on a novel 15.2M-pair corpus, comprising 10M caption-to-diagram pairs (MathCanvas-Imagen) and 5.2M step-by-step editing trajectories (MathCanvas-Edit), to master diagram generation and editing. Second, a Strategic Visual-Aided Reasoning stage fine-tunes the model on MathCanvas-Instruct, a new 219K-example dataset of interleaved visual-textual reasoning paths, teaching it when and how to leverage visual aids. To facilitate rigorous evaluation, we introduce MathCanvas-Bench, a challenging benchmark with 3K problems that require models to produce interleaved visual-textual solutions. Our model, BAGEL-Canvas, trained under this framework, achieves an 86% relative improvement over strong LMM baselines on MathCanvas-Bench, demonstrating excellent generalization to other public math benchmarks. Our work provides a complete toolkit-framework, datasets, and benchmark-to unlock complex, human-like visual-aided reasoning in LMMs. Project Page: https://mathcanvas.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
