Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using only the item content. Large Language Models (LLMs) represent a new frontier for this goal. The present research examines the feasibility of using an LLM to predict item difficulty for K-5 mathematics and reading assessment items (N = 5170). Two estimation approaches were implemented: (a) a direct estimation method that prompted the LLM to assign a single difficulty rating to each item, and (b) a feature-based strategy where the LLM extracted multiple cognitive and linguistic features, which were then used in ensemble tree-based models (random forests and gradient boosting) to predict difficulty. Overall, direct LLM estimates showed moderate to strong correlations with true item difficulties. However, their accuracy varied by grade level, often performing worse for early grades. In contrast, the feature-based method yielded stronger predictive accuracy, with correlations as high as r = 0.87 and lower error estimates compared to both direct LLM predictions and baseline regressors. These findings highlight the promise of LLMs in streamlining item development and reducing reliance on extensive field testing and underscore the importance of structured feature extraction. We provide a seven-step workflow for testing professionals who would want to implement a similar item difficulty estimation approach with their item pool.
A Multi-Agent Framework for Feature-Constrained Difficulty Control in Reading Comprehension Item Generation
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（韩国POSTECH人工智能研究生院）；Department of Computer Science and Engineering, POSTECH, Republic of Korea（韩国POSTECH计算机科学与工程系）
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
DIPF | Leibniz Institute for Research and Information in Education（德意志教育研究所 | 列支敦士登教育研究所）；Faculty of Computer Science, Goethe University Frankfurt（法兰克福歌德大学计算机学院）；Chemnitz University of Technology（化学工业大学）
Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks
School of Computing and Information, University of Pittsburgh（匹兹堡大学计算与信息学院）；Department of Educational Psychology, University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校教育心理学系）；Department of Emergency Medicine, University of Pittsburgh（匹兹堡大学急诊医学系）
Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（韩国立科学技术院人工智能研究生院）；Department of Computer Science and Engineering, POSTECH, Republic of Korea（韩国立科学技术院计算机科学与工程系）
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Maryland（马里兰大学）；Virginia Tech（弗吉尼亚理工大学）
Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院）；Arkansas State University（阿肯色州立大学）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；Heinz College of Information Systems and Public Policy, Carnegie Mellon University（卡内基梅隆大学海因茨信息系统与公共政策学院）；University of Memphis（孟菲斯大学）；Institute for Intelligent Systems, University of Memphis（孟菲斯大学智能系统研究所）；Department of Psychology, University of Memphis（孟菲斯大学心理学系）
NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty
Center for Language and Cognition, University of Groningen（语言与认知中心，格罗宁根大学）；Bernoulli Institute, University of Groningen（伯努利研究所，格罗宁根大学）；Department of Experimental Psychology, University of Groningen（实验心理学系，格罗宁根大学）
Latent Confidence Alignment for LLM Self-Assessment
Department of Information Management（信息管理系）；National Sun Yat-Sen University（国立中山大学）；Kaohsiung Medical University Hospital（高雄医学大学附设医院）；Kaohsiung Medical University（高雄医学大学）
Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence
Leverhulme Centre for the Future of Intelligence（勒沃休姆未来智能中心）；University of Cambridge（剑桥大学）；Valencian Research Institute for Artificial Intelligence（瓦伦西亚人工智能研究院）；Universitat Politècnica de València（瓦伦西亚理工大学）
