The rapid rise of large language models (LLMs) is reshaping the landscape of automatic assessment in education. While these systems demonstrate substantial advantages in adaptability to diverse question types and flexibility in output formats, they also introduce new challenges related to output uncertainty, stemming from the inherently probabilistic nature of LLMs. Output uncertainty is an inescapable challenge in automatic assessment, as assessment results often play a critical role in informing subsequent pedagogical actions, such as providing feedback to students or guiding instructional decisions. Unreliable or poorly calibrated uncertainty estimates can lead to unstable downstream interventions, potentially disrupting students' learning processes and resulting in unintended negative consequences. To systematically understand this challenge and inform future research, we benchmark a broad range of uncertainty quantification methods in the context of LLM-based automatic assessment. Although the effectiveness of these methods has been demonstrated in many tasks across other domains, their applicability and reliability in educational settings, particularly for automatic grading, remain underexplored. Through comprehensive analyses of uncertainty behaviors across multiple assessment datasets, LLM families, and generation control settings, we characterize the uncertainty patterns exhibited by LLMs in grading scenarios. Based on these findings, we evaluate the strengths and limitations of different uncertainty metrics and analyze the influence of key factors, including model families, assessment tasks, and decoding strategies, on uncertainty estimates. Our study provides actionable insights into the characteristics of uncertainty in LLM-based automatic assessment and lays the groundwork for developing more reliable and effective uncertainty-aware grading systems in the future.
Leveraging Generative AI for Enhancing Automated Assessment in Programming Education Contests
University of Bucharest Faculty of Mathematics and Computer Science(布加勒斯大学数学与计算机科学学院);Softbinator Technologies(Softbinator技术公司);Delft University of Technology(代尔夫特理工大学)
本文提出利用生成式 AI 自动生成编程竞赛高质量测试用例的 NLP 方法,并在 OJI、Kilonova.ro 和 IIOT 数据集上验证其可发现隐藏错误、减轻教师负担。
Using T-Norm Based Uncertainty Calculi in a Naval Situation Assessment Application
General Electric Corporate Research and Development(通用电气公司企业研究与开发中心)
RUM(Reasoning with Uncertainty Module)是一个基于KEE的集成软件工具,KEE是一个用面向对象语言实现的框架系统。RUM的体系结构由三层组成:表示层、推理层和控制层。表示层基于帧状数据结构,其捕获在推理层中使用的不确定性信息和在控制层中使用的不确定性元信息。推理层提供了五个基于T范数的不确定性演算的选择,用于执行信息的交集、分离、并集和池化。控制层使用元信息为每个上下文选择适当的演算,并解决信息中的最终无知或冲突。这一层还提供了一个上下文机制,使系统能够专注于知识库的相关部分,以及一个不确定性信念修正系统,该系统增量地更新非循环有向演绎图中的良构公式(wffs)的确定性值。RUM已经在海军和空中态势评估(SA)的一系列实验中进行了测试和验证,包括相关报告和跟踪,定位和分类平台,以及识别意图和威胁。最后给出了一个海上态势评估的实例。开发这些实验的测试平台环境由LOTTA提供,LOTTA是Flavors中实现的符号模拟器。该模拟器在多玩家对抗游戏中保持时变情况,玩家必须根据不确定和不完整的数据做出决策。RUM已被用于协助LOTTA玩家之一执行SA任务。