arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.02834cs.CL
收藏

CMM-Math:一个用于评估与增强多模态大模型数学推理能力的中文多模态数学数据集

CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models

School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);East China Normal University(华东师范大学);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室)

Wentao Liu, Qianjun Pan, Yi Zhang, Zhuo Liu, Ji Wu, Jie Zhou, Aimin Zhou, Qin Chen, Bo Jiang, Liang He

AI总结:

该研究发布含超2.8万样本的中文多模态数学数据集CMM-Math,同时提出三阶段训练的Math-LMM模型,在多数据集上验证其能有效提升多模态大模型的数学推理能力。

摘要:

大语言模型(LLMs)已在数学推理领域取得了可观的成果,而数学推理是人类智能的一项基础技能。过往多数研究都基于文本类数学推理数据集(如MATH、GSM8K)来提升并衡量大语言模型的性能。近年来,已有少量研究者发布了英文多模态数学数据集(如MATHVISTA和MATH-V),用于评估多模态大模型(LMMs)的效能。\n本文发布了一个包含基准测试部分与训练部分的中文多模态数学(CMM-Math)数据集,用于评估并增强多模态大模型的数学推理能力。CMM-Math包含超过28000个高质量样本,覆盖中国从小学到高中的12个年级,题型多样(如选择题、填空题等),且配有详细解答。特别之处在于,题目或选项中可能包含视觉上下文,这让该数据集更具挑战性。\n通过综合分析我们发现,当前最先进的多模态大模型在CMM-Math数据集上仍面临挑战,这凸显了进一步推进多模态大模型研发的必要性。我们还提出了一种多模态数学多模态大模型(Math-LMM),用于处理多图像与文本片段混合输入的问题。我们通过三个阶段训练该模型,分别是基础预训练、基础微调与数学微调。大量实验表明,在三个多模态数学数据集上与当前最优的多模态大模型相比,我们的模型有效提升了数学推理性能。

英文摘要:

Large language models (LLMs) have obtained promising results in mathematical reasoning, which is a foundational skill for human intelligence. Most previous studies focus on improving and measuring the performance of LLMs based on textual math reasoning datasets (e.g., MATH, GSM8K). Recently, a few researchers have released English multimodal math datasets (e.g., MATHVISTA and MATH-V) to evaluate the effectiveness of large multimodal models (LMMs). In this paper, we release a Chinese multimodal math (CMM-Math) dataset, including benchmark and training parts, to evaluate and enhance the mathematical reasoning of LMMs. CMM-Math contains over 28,000 high-quality samples, featuring a variety of problem types (e.g., multiple-choice, fill-in-the-blank, and so on) with detailed solutions across 12 grade levels from elementary to high school in China. Specifically, the visual context may be present in the questions or opinions, which makes this dataset more challenging. Through comprehensive analysis, we discover that state-of-the-art LMMs on the CMM-Math dataset face challenges, emphasizing the necessity for further improvements in LMM development. We also propose a Multimodal Mathematical LMM (Math-LMM) to handle the problems with mixed input of multiple images and text segments. We train our model using three stages, including foundational pre-training, foundational fine-tuning, and mathematical fine-tuning. The extensive experiments indicate that our model effectively improves math reasoning performance by comparing it with the SOTA LMMs over three multimodal mathematical datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑