arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.38402cs.CLcs.AI
收藏

作为医学试题难度预测的评判者与应试者的大语言模型

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

University of Florida(佛罗里达大学);National Commission on Certification of Physician Assistants(医师助理国家认证委员会)

Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

AI总结:

本研究利用开放权重LLMs作为评判者与模拟应试者,结合其产生的信号预测医学试题难度,在USMLE和PANCE上的斯皮尔曼相关系数分别达0.447和0.494,性能优于2024 BEA共享任务最优系统,可实用预测未发布试题难度。

摘要:

仅通过试题文本预测医学试题难度仍具挑战性。现有研究大多依赖从通用及领域特定语言模型中提取的语言特征或文本表示,但预测性能仍较为有限。本研究采用不同方法,利用开放权重大语言模型(LLMs)与试题交互时产生的难度相关信号。使用美国医师执照考试(USMLE)和医师助理国家委员会考试(PANCE)的医学试题,我们从两种角色考察LLMs:作为评判者,模型提供绝对难度评分和成对难度比较;作为模拟应试者,模型生成基于响应的信号,包括正确性、关键选项概率、响应熵及前两大选项的概率差。随后,我们将最强的评判信号与响应信号和基线试题特征结合,用于下游预测模型。成对难度评判是两个数据集中最一致的单个LLM信号,而模拟应试者信号与绝对评判在部分特征组合中提供互补信息。最优配置在USMLE上达到斯皮尔曼相关系数ρ=0.447,超过2024年BEA共享任务的最优系统(ρ=0.25),在PANCE上达到ρ=0.494。这些结果与专有前沿模型Claude Opus 5的直接绝对评判结果(分别为ρ=0.443和0.488)相当。总体而言,开放权重LLMs的评判与响应信号捕捉到了文本表示之外的难度相关信息,且由于这些模型可本地部署,该方法为预测未发布或高安全性试题的难度提供了实用选项。

英文摘要:

Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets and outperformed absolute ratings, and adding simulated-examinee signals yielded only small further gains. The best configurations reached a Spearman correlation of $ρ$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($ρ$ = .25), and $ρ$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($ρ$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑