As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation.
The multi-provider ensemble, consisted of open-weight models -- Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) -- and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members (kappa_w = 0.8652). The findings show a marked "Architecture-compatibility gap". Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a "Human-in-the-Loop" framework.
University of Bologna, Bologna, Italy(博洛尼亚大学);CNR - Institute of Cognitive Sciences and Technologies(意大利认知科学与技术研究院);University of Pisa, Pisa, Italy(比萨大学)
GAUSS: Benchmarking Structured Mathematical Skills for Large Language Models
Hyperbolic Labs;California Institute of Technology(加州理工学院);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Nvidia(英伟达);University of Washington(华盛顿大学);University of Hong Kong(香港大学)
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign(计算机科学系伊利诺伊大学厄巴纳-香槟分校);Department of Computer Science Stanford University(计算机科学系斯坦福大学)
FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
The Chinese University of Hong Kong(香港中文大学);Numina;Westlake University(西湖大学);M-A-P;University of California, Los Angeles(加州大学洛杉矶分校);Max Planck Institute for Intelligent Systems, Tübingen(蒂宾根马克斯·普朗克智能系统研究所)
Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany(机器学习与分析研究所(IMLA)、奥芬堡大学);University of Mannheim, Mannheim, Germany(曼海姆大学)
CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics
Academy of Mathematics and Systems Science, University of Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Sun Yat-sen University(中山大学);University of Cambridge(剑桥大学);East China Normal University(华东师范大学);Imperial College London(伦敦帝国理工学院);Stockholm Universitet(斯德哥尔摩大学);Numina;Moonshot AI
INTEGRALBENCH: Benchmarking LLMs with Definite Integral Problems
School of Software Engineering, Tongji University, Shanghai, China(同济大学软件工程学院);Polytechnic Institute, Zhejiang University, Zhejiang, China(浙江大学 polytechnic 院);ETH Zurich, Zurich, Switzerland(苏黎世联邦理工学院);Department of Computer Science(计算机科学系);Engineering, Hong Kong University of Science(科学大学工程学院);School of Mathematics(数学学院);Physics, Xi’an Jiaotong Liverpool University, Suzhou, China(西安交通大学利物浦大学物理学院)
Benchmarking Large Language Models with Integer Sequence Generation Tasks
Earth and Environmental Sciences Division Los Alamos National Laboratory(地球与环境科学 division 洛斯阿拉莫斯国家实验室);Theoretical Division Los Alamos National Laboratory(理论 division 洛斯阿拉莫斯国家实验室);Computational Sciences Division Los Alamos National Laboratory(计算科学 division 洛斯阿拉莫斯国家实验室)
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Teaching and Learning, Washington State University(华盛顿州立大学教学与学习系);College of Chemistry, Beijing Normal University(北京师范大学化学学院);Department of Computer Science, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);Department of Data Science, City University of Hong Kong(香港城市大学数据科学系)
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究室);Harbin Institute of Technology(哈尔滨工业大学);Central South University(中南大学);Fudan University(复旦大学);The University of Hong Kong(香港大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center(教育部文本计算与认知智能工程研究中心);Guizhou University(贵州大学)
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
Halıcıoğlu Data Science Institute, UC San Diego(哈里奇奥格卢数据科学研究所,加州大学圣地亚哥分校);Department of Computer Science and Engineering, UC San Diego(计算机科学与工程系,加州大学圣地亚哥分校);Department of Electrical Engineering, National Taiwan University(电气工程系,国立台湾大学);OpenAI
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
Technology Innovation Institute(技术创新研究院);University of Oslo(奥斯陆大学);Eötvös Loránd University(厄特沃什·罗兰大学);University of Guelma(盖勒马大学);The University of Manchester(曼彻斯特大学);Khalifa University(哈利法大学);Cyentific AS(赛恩蒂菲克公司)
RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
Artificial Intelligence Thrust, Information Hub, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China(人工智能 thrust,信息中心,香港科学与技术大学(广州),广州,中国)
PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
Peking University(北京大学);Mohamed bin Zayed University of Artificial Intelligence(莫扎伊德大学人工智能学院);National University of Singapore(新加坡国立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Science and Technology of China(中国科学技术大学);Cornell University(康奈尔大学);Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学)
MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Papers
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Gaotu Techedu Inc.(高途科技公司);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程研究中心,教育部);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);University of Science and Technology of China(中国科学技术大学)
VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Beijing Key Laboratory of Artificial Intelligence for Education(北京人工智能教育重点实验室);Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education(教育部智能技术与教育应用工程研究中心)
ATG: Benchmarking Automated Theorem Generation for Generative Language Models
Shenzhen Campus of Sun Yat-Sen University(中山大学深圳校区);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DarkMatter AI Research(DarkMatter AI研究)