多模态大语言模型(Multi-modal Large Language Models,MLLMs)凭借在视觉场景中的优异表现取得了显著进展,引发了前所未有的关注,但它们在视觉数学解题方面的能力尚未得到充分评估与清晰认知。我们经调研发现,现有基准中文字题目包含过多视觉内容,可能导致MLLMs无需真正解读输入图表就能推导答案。为此,我们推出MathVerse——一个旨在公平、深度评估MLLMs的综合性视觉数学基准。我们从公开渠道精心收集了2612道带图表的多学科高质量数学题,每道题由人工标注员转化为6个不同版本,各版本在多模态信息含量上存在差异,最终形成总计1.5万个测试样本。这种设计使MathVerse能够全面评估MLLMs是否以及在多大程度上能真正理解视觉图表以进行数学推理。此外,我们提出了思维链(Chain-of-Thought,CoT)评估策略,用于对输出答案进行细粒度评估:我们并非简单判断对错,而是采用GPT-4(V)自适应提取关键推理步骤,并对每个步骤打分并给出详细错误分析,从而揭示MLLMs的中间CoT推理质量。我们希望MathVerse基准能为指导MLLMs的未来发展提供独特洞见。项目页面:https://mathverse-cuhk.github.io
英文摘要:
The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io
Do Large Language Models Truly Grasp Mathematics? An Empirical Exploration From Cognitive Psychology
College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(科学、技术和研究局高性能计算研究所);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts
Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Nankai University(南开大学);Beijing Institute of Technology(北京理工大学);Baichuan Inc.(百度文心)
Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning
Zhejiang University(浙江大学);Fudan University(复旦大学);Soochow University(苏州大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models
University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学);National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学)
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Beijing University of Posts and Telecommunications(北京邮电大学);Wechat(腾讯微信);Tencent Inc.(腾讯公司);Huazhong University of Science and Technology(华中科技大学);Beijing Institute of Technology(北京理工大学)
PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Data Science & Artificial Intelligence Research Institute, China Unicom(中国unicom数据科学与人工智能研究院);Unicom Data Intelligence, China Unicom(中国unicom数据智能)
Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
University of Southern California(南加州大学);University of California San Diego(加州大学圣地亚哥分校);Carnegie Mellon University(卡内基梅隆大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs
Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所);Georgia Institute of Technology(佐治亚理工学院);Nanjing University of Science and Technology(南京理工大学);University of Oxford(牛津大学);The Ohio State University(俄亥俄州立大学)
GeoDANO: Geometric VLM with Domain Agnostic Vision Encoder
Graduate School of Artificial Intelligence, POSTECH(POSTECH人工智能研究生院);Department of Computer Science and Engineering, POSTECH(POSTECH计算机科学与工程系);Australian National University(澳大利亚国立大学)
NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generation
School of Intelligence Science and Technology, Nanjing University(智能科学与技术学院,南京大学);National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学)
Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?
Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港科学与技术大学);Shanghai Jiaotong University(上海交通大学);University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚实验室)
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
The Hong Kong Polytechnic University, Hong Kong, China(香港理工大学);ELLIS Institute Tübingen, Tübingen, Germany(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems, Tübingen, Germany(智能系统马克斯·普朗克研究所);Tübingen AI Center, Tübingen, Germany(图宾根人工智能中心);University of Surrey, Guildford, United Kingdom(萨里大学);The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA(北卡罗来纳大学教堂山分校)
Can Multimodal Large Language Models Truly Understand Small Objects?
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
AtomThink: Multimodal Slow Thinking with Atomic Step Reasoning
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);ETH Zurich(苏黎世联邦理工学院);Hong Kong University of Science and Technology(香港科技大学);University of Hong Kong(香港大学);Noah’s Ark Lab(诺亚实验室);Yinwang Intelligent Technology Co., Ltd.(亿纬智能科技有限公司)
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
Peking University(北京大学);University of Wisconsin - Madison(威斯康星大学麦迪逊分校);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);Zhongguancun Laboratory(中关村实验室);Engineering Research Center of Information Networks(信息网络工程研究中心)