Estimating the difficulty of multiple-choice questions would be great help for educators who must spend substantial time creating and piloting stimuli for their tests, and for learners who want to practice. Supervised approaches to difficulty estimation have yielded to date mixed results. In this contribution we leverage an aspect of generative large models which might be seen as a weakness when answering questions, namely their uncertainty, and exploit it towards exploring correlations between two different metrics of uncertainty, and the actual student response distribution. While we observe some present but weak correlations, we also discover that the models' behaviour is different in the case of correct vs wrong answers, and that correlations differ substantially according to the different question types which are included in our fine-grained, previously unused dataset of 451 questions from a Biopsychology course. In discussing our findings, we also suggest potential avenues to further leverage model uncertainty as an additional proxy for item difficulty.
A Multi-Agent Framework for Feature-Constrained Difficulty Control in Reading Comprehension Item Generation
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea(韩国POSTECH人工智能研究生院);Department of Computer Science and Engineering, POSTECH, Republic of Korea(韩国POSTECH计算机科学与工程系)
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
Mendoza College of Business, University of Notre Dame(圣母大学门多萨商学院);College of Information and Computer Sciences, UMass Amherst(马萨诸塞大学阿默斯特分校信息与计算机科学学院);Department of Psychology and Human Development, Vanderbilt University(范德堡大学心理学与人类发展系);Department of Computer Science, UMass Lowell(马萨诸塞大学洛厄尔分校计算机科学系);Center for Healthcare Organization & Implementation Research, Bedford VA(贝德福德退伍军人事务部医疗保健组织与实施研究中心)
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
DIPF | Leibniz Institute for Research and Information in Education(德意志教育研究所 | 列支敦士登教育研究所);Faculty of Computer Science, Goethe University Frankfurt(法兰克福歌德大学计算机学院);Chemnitz University of Technology(化学工业大学)
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Maryland(马里兰大学);Virginia Tech(弗吉尼亚理工大学)
Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea(韩国立科学技术院人工智能研究生院);Department of Computer Science and Engineering, POSTECH, Republic of Korea(韩国立科学技术院计算机科学与工程系)
Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks
School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院);Department of Educational Psychology, University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校教育心理学系);Department of Emergency Medicine, University of Pittsburgh(匹兹堡大学急诊医学系)
Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?
RIKEN;The University of Osaka(大阪大学);Nara Institute of Science and Technology(奈良科学技術大學);National Tsing Hua University(國立清華大學);The University of Tokyo(東京大學);Tohoku University(東北大學)
Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院);Arkansas State University(阿肯色州立大学);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院);Heinz College of Information Systems and Public Policy, Carnegie Mellon University(卡内基梅隆大学海因茨信息系统与公共政策学院);University of Memphis(孟菲斯大学);Institute for Intelligent Systems, University of Memphis(孟菲斯大学智能系统研究所);Department of Psychology, University of Memphis(孟菲斯大学心理学系)
Latent Confidence Alignment for LLM Self-Assessment
Department of Information Management(信息管理系);National Sun Yat-Sen University(国立中山大学);Kaohsiung Medical University Hospital(高雄医学大学附设医院);Kaohsiung Medical University(高雄医学大学)