arXiv 2505.134272025-06-05cs.AIcs.CV

MM-PRM：通过可扩展的步骤级监督增强多模态数学推理

MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision

Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）

Lingxiao Du, Fanqing Meng, Zongkai Liu, Zhixiang Zhou, Ping Luo, Qiaosheng Zhang, Wenqi Shao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 MM-PRM，通过自动化 MCTS 流程构建大规模步骤级监督数据训练过程奖励模型，用于多模态数学推理路径评分，显著提升域内外基准上的逻辑鲁棒性。

摘要：

尽管多模态大语言模型（MLLMs）在视觉-语言理解方面取得了令人瞩目的进展，但它们在复杂的多步推理任务中仍然表现不佳，常常产生逻辑不一致或仅部分正确的解答。一个关键限制在于缺乏对中间推理步骤的细粒度监督。为了解决这一问题，我们提出了 MM-PRM，这是一个在完全自动化、可扩展的框架中训练的过程奖励模型。我们首先构建了 MM-Policy，这是一个在多样化数学推理数据上训练的强大多模态模型。然后，我们构建了 MM-K12，这是一个包含 10,000 道具有可验证答案的多模态数学问题的精选数据集，用作种子数据。利用基于蒙特卡洛树搜索（MCTS）的流程，我们在无需人工标注的情况下生成了超过 700k 条步骤级标注。由此得到的过程奖励模型（PRM）被用于在 Best-of-N 推理设置中对候选推理路径进行评分，并在域内（MM-K12 测试集）和域外（OlympiadBench、MathVista 等）基准上均取得了显著提升。进一步的分析证实了软标签、较小的学习率以及路径多样性在优化 PRM 性能方面的有效性。MM-PRM 表明，过程监督是增强多模态推理系统逻辑鲁棒性的有力工具。我们在 https://github.com/ModalMinds/MM-PRM 发布了所有代码和数据。

英文摘要：

While Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language understanding, they still struggle with complex multi-step reasoning, often producing logically inconsistent or partially correct solutions. A key limitation lies in the lack of fine-grained supervision over intermediate reasoning steps. To address this, we propose MM-PRM, a process reward model trained within a fully automated, scalable framework. We first build MM-Policy, a strong multimodal model trained on diverse mathematical reasoning data. Then, we construct MM-K12, a curated dataset of 10,000 multimodal math problems with verifiable answers, which serves as seed data. Leveraging a Monte Carlo Tree Search (MCTS)-based pipeline, we generate over 700k step-level annotations without human labeling. The resulting PRM is used to score candidate reasoning paths in the Best-of-N inference setup and achieves significant improvements across both in-domain (MM-K12 test set) and out-of-domain (OlympiadBench, MathVista, etc.) benchmarks. Further analysis confirms the effectiveness of soft labels, smaller learning rates, and path diversity in optimizing PRM performance. MM-PRM demonstrates that process supervision is a powerful tool for enhancing the logical robustness of multimodal reasoning systems. We release all our codes and data at https://github.com/ModalMinds/MM-PRM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。