MAmmoTH-VL：通过大规模指令微调激发多模态推理能力
MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
AI总结：
本文提出MAmmoTH-VL方法，通过构建包含1200万条带中间推理过程的多模态指令微调数据集，显著提升了开源多模态大语言模型的推理能力，并在多个基准上取得最先进性能。
摘要：
开源多模态大语言模型（MLLMs）在广泛的多模态任务中展现出显著潜力。然而，其推理能力仍受限于现有的指令微调数据集，这些数据集主要从VQA、AI2D和ChartQA等学术数据集改造而来。这些数据集针对的是简单任务，且仅提供短语级答案，不包含任何中间推理过程。为解决这些挑战，我们提出了一种可扩展且成本高效的方法，用于构建一个大规模多模态指令微调数据集，该数据集包含丰富的中间推理过程，旨在激发链式思维（CoT）推理能力。仅使用开放模型，我们创建了一个包含1200万条指令-响应对的数据集，覆盖多样化的推理密集型任务，并提供详细且忠实的推理过程。实验表明，在该数据集上训练MLLMs可显著提升推理能力，在MathVerse（+8.1%）、MMMU-Pro（+7%）和MuirBench（+13.3%）等基准上取得了最先进的性能。此外，该模型在非推理类基准上也表现出高达4%的显著提升。消融研究进一步凸显了数据集构建过程中重写和自我过滤等关键组件的重要性。
英文摘要：
Open-source multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. However, their reasoning capabilities remain constrained by existing instruction-tuning datasets, which were predominately repurposed from academic datasets such as VQA, AI2D, and ChartQA. These datasets target simplistic tasks, and only provide phrase-level answers without any intermediate rationales. To address these challenges, we introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales designed to elicit CoT reasoning. Using only open models, we create a dataset containing 12M instruction-response pairs to cover diverse, reasoning-intensive tasks with detailed and faithful rationales. Experiments demonstrate that training MLLMs on this dataset significantly improves reasoning capabilities, achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%), MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation studies further highlight the importance of key components, such as rewriting and self-filtering, in the dataset construction process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
