arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.17294cs.CL
收藏

Math-LLaVA:为多模态大语言模型自举提升数学推理能力

Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models

University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学);National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学)

Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, Roy Ka-Wei Lee

AI总结:

针对现有开源多模态数学微调数据不足的问题,本研究构建了包含36万条问答对的MathV360K数据集,基于LLaVA-1.5微调得到的Math-LLaVA在多个数学推理基准上性能显著提升,泛化性也得到增强。

摘要:

大语言模型(LLMs)已展现出令人瞩目的推理能力,尤其是在文本类数学问题求解方面。然而,现有的开源图像指令微调数据集每张图像包含的问答对数量有限,未能充分利用视觉信息来增强多模态大语言模型(MLLMs)的多模态数学推理能力。为弥补这一缺口,我们针对高质量、多样化多模态数学数据集匮乏的问题,从24个现有数据集中收集了4万张带问答对的高质量图像,并合成了32万条新问答对,构建了MathV360K数据集,该数据集同时提升了多模态数学问题的广度与深度。我们提出了Math-LLaVA,这是一个基于LLaVA-1.5、使用MathV360K微调的模型。这一新颖方法显著提升了LLaVA-1.5的多模态数学推理能力,在MathVista的minitest分割集上得分提升19分,性能与GPT-4V相当,且在Math-V与MathVerse上取得领先表现。此外,Math-LLaVA展现出更强的泛化性,在MMMU基准测试上实现了大幅提升。我们的研究凸显了数据集多样性与合成对推进MLLMs数学推理能力的重要性。代码与数据可在以下网址获取:https://github.com/HZQ950419/Math-LLaVA。

英文摘要:

Large language models (LLMs) have demonstrated impressive reasoning capabilities, particularly in textual mathematical problem-solving. However, existing open-source image instruction fine-tuning datasets, containing limited question-answer pairs per image, do not fully exploit visual information to enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs (MLLMs). To bridge this gap, we address the lack of high-quality, diverse multimodal mathematical datasets by collecting 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs, creating the MathV360K dataset, which enhances both the breadth and depth of multimodal mathematical questions. We introduce Math-LLaVA, a LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach significantly improves the multimodal mathematical reasoning capabilities of LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V on MathVista's minitest split, and yielding leading performance on Math-V and MathVerse. Furthermore, Math-LLaVA demonstrates enhanced generalizability, showing substantial improvements on the MMMU benchmark. Our research highlights the importance of dataset diversity and synthesis in advancing MLLMs' mathematical reasoning abilities. The code and data are available at: \url{https://github.com/HZQ950419/Math-LLaVA}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑