The use of Large Language Models (LLMs) in mathematical reasoning has become a cornerstone of related research, demonstrating the intelligence of these models and enabling potential practical applications through their advanced performance, such as in educational settings. Despite the variety of datasets and in-context learning algorithms designed to improve the ability of LLMs to automate mathematical problem solving, the lack of comprehensive benchmarking across different datasets makes it complicated to select an appropriate model for specific tasks. In this project, we present a benchmark that fairly compares seven state-of-the-art in-context learning algorithms for mathematical problem solving across five widely used mathematical datasets on four powerful foundation models. Furthermore, we explore the trade-off between efficiency and performance, highlighting the practical applications of LLMs for mathematical reasoning. Our results indicate that larger foundation models like GPT-4o and LLaMA 3-70B can solve mathematical reasoning independently from the concrete prompting strategy, while for smaller models the in-context learning approach significantly influences the performance. Moreover, the optimal prompt depends on the chosen foundation model. We open-source our benchmark code to support the integration of additional models in future research.
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen(法律与人工智能教授、人工智能与法律研究所主任,图宾根大学)
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations(国际关系学院);Kedge Business School(凯致商学院);Peking University(北京大学);Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Capital Normal University(首都师范大学)
Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究 institute(I2R),A*STAR,新加坡);Centre for Frontier AI Research (CFAR), A*STAR, Singapore(前沿人工智能研究 centre(CFAR),A*STAR,新加坡)
RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios
Center for Chinese Linguistics, Department of Chinese Language and Literature, Peking University(北京大学中文语言文学系中文语言学中心);School of Computer Science, State Key Laboratory of Multimedia Information Processing, Peking University(北京大学计算机学院多媒体信息处理国家重点实验室);Huawei Noah’s Ark Lab, China(华为诺亚实验室,中国)
Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands(医学信息学系,阿姆斯特丹大学医学中心,阿姆斯特丹大学,阿姆斯特丹,荷兰);Amsterdam Public Health, Methodology, Amsterdam, The Netherlands(阿姆斯特丹公共健康,方法学,阿姆斯特丹,荷兰)
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tongyi Lab, Alibaba Group(阿里云实验室)
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)