arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.14624cs.CVcs.AIcs.CLcs.LG
收藏

MathVerse:你的多模态大语言模型真的看懂了视觉数学题中的图表吗?

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);CUHK(香港中文大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li

AI总结:

针对现有视觉数学基准易让多模态大语言模型靠文字信息蒙题的问题,研究团队构建了含1.5万多样本的MathVerse基准,并提出细粒度CoT评估策略,以深度评测模型对图表的真实理解能力。

摘要:

多模态大语言模型(Multi-modal Large Language Models,MLLMs)凭借在视觉场景中的优异表现取得了显著进展,引发了前所未有的关注,但它们在视觉数学解题方面的能力尚未得到充分评估与清晰认知。我们经调研发现,现有基准中文字题目包含过多视觉内容,可能导致MLLMs无需真正解读输入图表就能推导答案。为此,我们推出MathVerse——一个旨在公平、深度评估MLLMs的综合性视觉数学基准。我们从公开渠道精心收集了2612道带图表的多学科高质量数学题,每道题由人工标注员转化为6个不同版本,各版本在多模态信息含量上存在差异,最终形成总计1.5万个测试样本。这种设计使MathVerse能够全面评估MLLMs是否以及在多大程度上能真正理解视觉图表以进行数学推理。此外,我们提出了思维链(Chain-of-Thought,CoT)评估策略,用于对输出答案进行细粒度评估:我们并非简单判断对错,而是采用GPT-4(V)自适应提取关键推理步骤,并对每个步骤打分并给出详细错误分析,从而揭示MLLMs的中间CoT推理质量。我们希望MathVerse基准能为指导MLLMs的未来发展提供独特洞见。项目页面:https://mathverse-cuhk.github.io

英文摘要:

The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑