ELMES: An Automated Framework for Evaluating Large Language Models in Educational Scenarios
Shanghai Innavation Institute（上海创新研究院）；Shanghai Institute of AI for Education（上海人工智能教育研究院）；Department of Educational Information Technology（教育信息技术系）；School of Computer Science and Technology（计算机科学与技术学院）
The emergence of Large Language Models (LLMs) presents transformative opportunities for education, generating numerous novel application scenarios. However, significant challenges remain: evaluation metrics vary substantially across different educational scenarios, while many emerging scenarios lack appropriate assessment metrics. Current benchmarks predominantly measure general intelligence rather than pedagogical capabilities. To address this gap, we introduce ELMES, an open-source automated evaluation framework specifically designed for assessing LLMs in educational settings. ELMES features a modular architecture that enables researchers to create dynamic, multi-agent dialogues through simple configuration files, facilitating flexible scenario design without requiring extensive programming expertise. The framework incorporates a hybrid evaluation engine that objectively quantifies traditionally subjective pedagogical metrics using an LLM-as-a-Judge methodology. We conduct systematic benchmarking of state-of-the-art LLMs across four critical educational scenarios: Knowledge Point Explanation, Guided Problem-Solving Teaching, Interdisciplinary Lesson Plan Generation, and Contextualized Question Generation, employing fine-grained metrics developed in collaboration with education specialists. Our results demonstrate distinct capability distributions among models, revealing context-specific strengths and limitations. ELMES provides educators and researchers with an accessible evaluation framework that significantly reduces adaptation barriers for diverse educational applications while advancing the practical implementation of LLMs in pedagogy. The framework is publicly available at \emph{https://github.com/sii-research/elmes.git}.
OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models
Chosun University（chosun大学）；Korea University（韩国大学）；Ewha Womans University（成均馆大学）；Korea Institute for Curriculum and Evaluation（韩国课程评价院）；Seoul National University（首尔国立大学）；Texas A&M University（德克萨斯农工大学）；Indiana University Bloomington（印第安纳大学布卢明顿分校）
A Report on the llms evaluating the high school questions
School of Environmental Science and Engineering, Guangdong University of Technology（广东工业大学环境科学与工程学院）；School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）
East China Normal University（东华师范大学）；Zhejiang University（浙江大学）；Imperial College London（伦敦帝国学院）
With the rapid development of large language models (LLMs), various LLM-based works have been widely applied in educational fields. However, most existing LLMs and their benchmarks focus primarily on the knowledge dimension, largely neglecting the evaluation of cultivation capabilities that are essential for real-world educational scenarios. Additionally, current benchmarks are often limited to a single subject or question type, lacking sufficient diversity. This issue is particularly prominent within the Chinese context. To address this gap, we introduce OmniEduBench, a comprehensive Chinese …
EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
University of Bologna（博洛尼亚大学）；Villa i Tatti – The Harvard University Center for Italian Renaissance Studies（哈佛大学意大利文艺复兴研究中心（I Tatti））；University of Copenhagen（哥本哈根大学）
SHAPE: Unifying Safety, Helpfulness and Pedagogy for Educational LLMs
Center for Data Science, New York University Shanghai（纽约大学上海数据科学中心）；Courant Institute of Mathematical Sciences, New York University（纽约大学Courant数学科学研究所）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models
East China Normal University（华东师范大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Education University of Hong Kong（香港教育大学）
The use of large language models to enhance cancer clinical trial educational materials
Massachusetts Institute of Technology（麻省理工学院）；Technical University of Munich（慕尼黑工业大学）；Mass General Brigham（麻省总医院布里格姆医疗系统）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Centaur Labs（半人马实验室）；Digidence；Loyola University Chicago（芝加哥洛约拉大学）
EduResearchBench: A Hierarchical Atomic Task Decomposition Benchmark for Full-Lifecycle Educational Research
Shanghai Institute of AI for Education, East China Normal University（上海人工智能教育研究院，东华大学）；Shanghai Innovation Institute（上海创新研究院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）
Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
Quantic School of Business and Technology（Quantic商业与技术学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Indonesia University of Education（印度尼西亚教育大学）；Monash University（莫纳什大学）
ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities
School of Psychology, South China Normal University（南方科技大学心理学院）；Information Hub, Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）信息中心）；School of AI, Guangzhou University（广州大学人工智能学院）；College of Cyber Security, Jinan University（济南大学网络安全学院）
Where Should I Study? Biased Language Models Decide! Evaluating Fairness in LMs for Academic Recommendations
Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI)（负责任人工智能中心（CeRAI）、Wadhwani数据科学与人工智能学校（WSAI））；Indian Institute of Technology Madras（印度理工学院马德拉斯学院）
