CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);East China Normal University(华东师范大学);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室)
Wentao Liu, Qianjun Pan, Yi Zhang, Zhuo Liu, Ji Wu, Jie Zhou, Aimin Zhou, Qin Chen, Bo Jiang, Liang He
Large language models (LLMs) have obtained promising results in mathematical reasoning, which is a foundational skill for human intelligence. Most previous studies focus on improving and measuring the performance of LLMs based on textual math reasoning datasets (e.g., MATH, GSM8K). Recently, a few researchers have released English multimodal math datasets (e.g., MATHVISTA and MATH-V) to evaluate the effectiveness of large multimodal models (LMMs). In this paper, we release a Chinese multimodal math (CMM-Math) dataset, including benchmark and training parts, to evaluate and enhance the mathematical reasoning of LMMs. CMM-Math contains over 28,000 high-quality samples, featuring a variety of problem types (e.g., multiple-choice, fill-in-the-blank, and so on) with detailed solutions across 12 grade levels from elementary to high school in China. Specifically, the visual context may be present in the questions or opinions, which makes this dataset more challenging. Through comprehensive analysis, we discover that state-of-the-art LMMs on the CMM-Math dataset face challenges, emphasizing the necessity for further improvements in LMM development. We also propose a Multimodal Mathematical LMM (Math-LMM) to handle the problems with mixed input of multiple images and text segments. We train our model using three stages, including foundational pre-training, foundational fine-tuning, and mathematical fine-tuning. The extensive experiments indicate that our model effectively improves math reasoning performance by comparing it with the SOTA LMMs over three multimodal mathematical datasets.
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community(多模态艺术投影研究共同体);Hong Kong University of Science and Technology(香港科技大学);University of Waterloo(滑铁卢大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peking University(北京大学);The Hong Kong Polytechnic University(香港理工大学);Waseda University(早稻田大学);University of Manchester(曼彻斯特大学)
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC
TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicine
School of Artificial Intelligence in Traditional Chinese Medicine, Shanghai University of Traditional Chinese Medicine, Shanghai, China(上海中医药大学人工智能学院);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);University of Washington, Seattle, Washington, US(华盛顿大学)
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
ShanghaiTech University(上海科技大学);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心);University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Center for Processor Technologies(上海处理器技术创新中心);Shanghai Jiao Tong University(上海交通大学);National Engineering Laboratory for Brain-inspired Intelligence Technology and Application(国家类脑智能技术及应用工程实验室);Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education(脑与认知科学教育部重点实验室)
OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院);School of Computer and Artificial Intelligence, Zhengzhou University(郑州大学计算机与人工智能学院)
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);East China Normal University(华东师范大学);Oxford(牛津大学);University of Sydney(悉尼大学);Yimudata(云墨数据)
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所);Applied AI Institute(应用人工智能研究所);Charles University(查尔斯大学);Nazarbayev University(纳扎尔拜大学);Inria(法国国家信息与自动化研究所);Indian Institute of Technology, Kharagpur (IIT Kharagpur)(印度Kharagpur理工学院);German University of Digital Science(德国数字科学大学)
Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
Lehigh University(里海大学);Squirrel Ai Learning(松鼠AI学习);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Michigan State University(密歇根州立大学)