MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；ETH AI Center（ETH人工智能中心）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（达姆施塔特技术大学计算机科学系）；Professorship for Learning Sciences and Higher Education, ETH Zurich（苏黎世联邦理工学院学习科学与高等教育教授职位）
Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, Mrinmaya Sachan
Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.
LLMs Are Already Good Tutors: Training-Free Prompt Optimization for Pedagogical Math Tutoring
Korea University Sejong Campus（韩国大学世宗校区）；Gyeonggi Institute of Education（京畿教育学院）；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Opentutorials；Chosun University（全州大学）；Korea University Korean Studies Center（韩国大学韩学研究中心）
Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure
ETH Zurich（苏黎世联邦理工学院）；École polytechnique（巴黎综合理工学院）；ETH AI Center（苏黎世联邦理工学院AI中心）；Professorship for Learning Sciences and Higher Education, ETH Zurich（苏黎世联邦理工学院学习科学与高等教育教授席位）
SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems
Eindhoven University of Technology（埃因霍温理工大学）；Indian Institute of Technology Kharagpur（印度理工学院Khargpur分校）；Cisco Research（思科研究）；Taras Shevchenko National University of Kyiv（塔拉斯·谢甫琴科基辅国立大学）；National Technical University of Ukraine（乌克兰国家技术大学）；New York University（纽约大学）
Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors
Washington University in St. Louis（圣路易斯华盛顿大学）；Department of Operations Research and Engineering Management, Southern Methodist University（南卫理公会大学运筹学与工程管理系）
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University（昌原大学）；Korea University（韩国大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程与评价研究院）；Upstage；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Nanyang Technological University（南洋理工大学）
MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems
ETH AI Center（苏黎世联邦理工学院人工智能中心）；ETH Zurich（苏黎世联邦理工学院）；TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；National Institute of Education, Nanyang Technological University（南洋理工大学国立教育学院）
How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction
Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学人工智能教育实验室）
Developing a Tutoring Dialog Dataset to Optimize LLMs for Educational Use
Graduate School of Information Science and Electrical Engineering（信息科学与电气工程研究生院）；Kyushu University（九州大学）；Faculty of Information Science and Electrical Engineering（信息科学与电气工程学部）
ETH AI Center（苏黎世联邦理工学院人工智能中心）；The Chinese University of Hong Kong（香港中文大学）；ETH Zürich（苏黎世联邦理工学院）；TU Darmstadt（达姆施塔特工业大学）
Designing dialog tutors has been challenging as it involves modeling the diverse and complex pedagogical strategies employed by human tutors. Although there have been significant recent advances in neural conversational systems using large language models (LLMs) and growth in available dialog corpora, dialog tutoring has largely remained unaffected by these advances. In this paper, we rigorously analyze various generative language models on two dialog tutoring datasets for language learning using automatic and human evaluations to understand the new opportunities brought by these advances as w…
From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；ETH AI Center（ETH人工智能中心）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（达姆施塔特技术大学计算机科学系、应用网络安全国家研究中心ATHENE）
