仅通过试题文本预测医学试题难度仍具挑战性。现有研究大多依赖从通用及领域特定语言模型中提取的语言特征或文本表示,但预测性能仍较为有限。本研究采用不同方法,利用开放权重大语言模型(LLMs)与试题交互时产生的难度相关信号。使用美国医师执照考试(USMLE)和医师助理国家委员会考试(PANCE)的医学试题,我们从两种角色考察LLMs:作为评判者,模型提供绝对难度评分和成对难度比较;作为模拟应试者,模型生成基于响应的信号,包括正确性、关键选项概率、响应熵及前两大选项的概率差。随后,我们将最强的评判信号与响应信号和基线试题特征结合,用于下游预测模型。成对难度评判是两个数据集中最一致的单个LLM信号,而模拟应试者信号与绝对评判在部分特征组合中提供互补信息。最优配置在USMLE上达到斯皮尔曼相关系数ρ=0.447,超过2024年BEA共享任务的最优系统(ρ=0.25),在PANCE上达到ρ=0.494。这些结果与专有前沿模型Claude Opus 5的直接绝对评判结果(分别为ρ=0.443和0.488)相当。总体而言,开放权重LLMs的评判与响应信号捕捉到了文本表示之外的难度相关信息,且由于这些模型可本地部署,该方法为预测未发布或高安全性试题的难度提供了实用选项。
英文摘要:
Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets and outperformed absolute ratings, and adding simulated-examinee signals yielded only small further gains. The best configurations reached a Spearman correlation of $ρ$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($ρ$ = .25), and $ρ$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($ρ$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Department of Artificial Intelligence, Chung-Ang University, Seoul, Republic of Korea(Chung-Ang 大学人工智能系);Department of Industrial Engineering, Seoul National University, Seoul, Republic of Korea(首尔国立大学工业工程系)
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Trismik;Leverhulme Centre for the Future of Intelligence, University of Cambridge(Leverhulme智能未来研究中心,剑桥大学);Universitat Politècnica de València(瓦伦西亚理工大学);University of Cambridge(剑桥大学)
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
Estimating Item Difficulty with Large Language Models as Experts
Department of Methodology and Statistics, Tilburg University(蒂尔堡大学方法学与统计学系);Smart Business Technologies(智能商务技术公司);Department of Psychological Methods, University of Amsterdam(阿姆斯特丹大学心理方法系);Prowise Learn, Amsterdam(Prowise Learn公司,阿姆斯特丹);Department of Information and Computing Sciences, Utrecht University(乌得勒支大学信息与计算科学系)
Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Mathematical and Natural Sciences, Arizona State University(亚利桑那州立大学数学与自然科学学院)
Artificiality's First Exam: An LLM-Referenced C-test for Supraintelligence
Leverhulme Centre for the Future of Intelligence(勒沃休姆未来智能中心);University of Cambridge(剑桥大学);Valencian Research Institute for Artificial Intelligence(瓦伦西亚人工智能研究院);Universitat Politècnica de València(瓦伦西亚理工大学)
Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Columbia University(哥伦比亚大学);Johns Hopkins University(约翰斯·霍普金斯大学);The University of Chicago(芝加哥大学);The Pennsylvania State University(宾夕法尼亚州立大学);The Hong Kong Polytechnic University(香港理工大学)
Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Shanghai Jiao Tong University(上海交通大学);Arizona State University(亚利桑那州立大学);Eastern Institute of Technology(东方理工学院)
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
National Laboratory for Scientific Computing (LNCC)(国家科学计算实验室);Institute of Artificial Intelligence(人工智能研究所)
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Salesforce AI Research(Salesforce AI 研究院);University of Cambridge(剑桥大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Carnegie Mellon University(卡内基梅隆大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)