How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues, simulate how different agents would respond to a student, and compare these counterpart responses in terms of three abilities: speak like a teacher, understand a student, help a student. Our method builds on the reliability of comparative judgments in education and uses a probabilistic model and Bayesian sampling to infer estimates of pedagogical ability. We find that, even though conversational agents (Blender in particular) perform well on conversational uptake, they are quantifiably worse than real teachers on several pedagogical dimensions, especially with regard to helpfulness (Blender: Δ ability = -0.75; GPT-3: Δ ability = -0.93).
AIriskEval-edu Demo: Auditing of Pedagogical Risks in Educational Explanations
BiometricsAI, Universidad Autónoma de Madrid (UAM)（生物识别人工智能，马德里自治大学（UAM））；GHIA, Universidad Autónoma de Madrid (UAM)（GHIA，马德里自治大学（UAM））；Universidad de Las Palmas de Gran Canaria (ULPGC)（大加那利岛拉斯帕尔马斯大学（ULPGC））
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University（昌原大学）；Korea University（韩国大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程与评价研究院）；Upstage；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Nanyang Technological University（南洋理工大学）
Higher School of Sciences and Techniques of Tunis（突尼斯高等科学与技术学院）；University of Monastir（莫纳斯提尔大学）
In this article we focus firstly on the principle of pedagogical indexing and characteristics of Arabic language and secondly on the possibility of adapting the standard for describing learning resources used (the LOM and its Application Profiles) with learning conditions such as the educational levels of students and their levels of understanding,... the educational context with taking into account the representative elements of text, text length, ... in particular, we put in relief the specificity of the Arabic language which is a complex language, characterized by its flexion, its voyellati…
Pedagogical Alignment for Vision-Language-Action Models: A Comprehensive Framework for Data, Architecture, and Evaluation in Education
Chosun University（全州大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程评价院）；Nanyang Technological University（南洋理工大学）
SHAPE: Unifying Safety, Helpfulness and Pedagogy for Educational LLMs
Center for Data Science, New York University Shanghai（纽约大学上海数据科学中心）；Courant Institute of Mathematical Sciences, New York University（纽约大学Courant数学科学研究所）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations
Cátedra ENIA UAM-VERIDAS en IA Responsable（ENIA UAM-VERIDAS负责任人工智能教席）；NextGenerationEU PRTR（下一代欧盟复苏计划）；MICIU/FEDER（西班牙科学、创新与大学部/欧洲区域发展基金）；MICIU/AEI（西班牙科学、创新与大学部/国家研究机构）；Comunidad de Madrid（马德里自治区）；MINECO/FEDER（西班牙经济与竞争力部/欧洲区域发展基金）
ELMES: An Automated Framework for Evaluating Large Language Models in Educational Scenarios
Shanghai Innavation Institute（上海创新研究院）；Shanghai Institute of AI for Education（上海人工智能教育研究院）；Department of Educational Information Technology（教育信息技术系）；School of Computer Science and Technology（计算机科学与技术学院）
When learning a second language (L2), one of the most important but tedious components that often demoralizes students with its ineffectiveness and inefficiency is vocabulary acquisition, or more simply put, memorizing words. In light of such, a personalized and educational vocabulary recommendation system that traces a learner's vocabulary knowledge state would have an immense learning impact as it could resolve both issues. Therefore, in this paper, we propose and release data for a novel task called Pedagogical Word Recommendation (PWR). The main goal of PWR is to predict whether a given le…
University of Southern Queensland（南昆士兰大学）；Lingnan University（岭南大学）
Sentiment analysis AKA opinion mining is one of the most widely used NLP applications to identify human intentions from their reviews. In the education sector, opinion mining is used to listen to student opinions and enhance their learning-teaching practices pedagogically. With advancements in sentiment annotation techniques and AI methodologies, student comments can be labelled with their sentiment orientation without much human intervention. In this review article, (1) we consider the role of emotional analysis in education from four levels: document level, sentence level, entity level, and …
LLM Agents for Education: Advances and Applications
Squirrel Ai Learning；Fudan University（复旦大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators' time and benefiting student learning. Yet, QG systems have not been widely adopted in classrooms to date. In this work, we aim to pinpoint key impediments and investigate how to improve the usability of automatic QG techniques for educational purposes by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models. We perform an in-depth need finding study with 11 instructors across 7 different universities, and summarize their …
The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues
KU Leuven（鲁汶大学）；imec（比利时微电子研究中心）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；King’s College London（伦敦国王学院）；Universidad Central del Ecuador（厄瓜多尔中央大学）；Stanford University（斯坦福大学）
