衡量重要性——或者便利性？：基于LLM的评分系统对无关因素的鲁棒性
Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors
AI总结：
研究探讨了基于LLM的评分系统在面对无关因素时的鲁棒性，发现其在处理无意义填充文本、拼写错误等时表现稳健，但对偏离主题的回答惩罚较重，为未来设计更相关的LLM评分系统提供了支持。
摘要：
自动化系统在教育测试行业广泛应用，用于开放性评估和作文评分。这些系统通常能与训练过的评分者表现相当或更优，但常被证明易受无关因素影响。随着大型语言模型在自动化评分中的使用增加，对“幻觉”和这些LLM评分方法对无关因素鲁棒性的关注再次增强。本研究探讨了无关因素对双架构LLM评分系统的影响，该系统用于评估情境判断测试中的短作文类开放性问题。研究发现，评分系统在处理无意义填充文本、拼写错误和写作复杂性时总体稳健。复制大量文本导致系统预测分数较低，与以往非LLM评分系统研究结果相悖，而偏离主题的回答被系统严重惩罚。这些结果为未来设计更相关的LLM评分系统提供了积极支持。
英文摘要：
Automated systems have been widely adopted across the educational testing industry for open-response assessment and essay scoring. These systems commonly achieve performance levels comparable to or superior than trained human raters, but have frequently been demonstrated to be vulnerable to the influence of construct-irrelevant factors (i.e., features of responses that are unrelated to the construct assessed) and adversarial conditions. Given the rising usage of large language models in automated scoring systems, there is a renewed focus on ``hallucinations'' and the robustness of these LLM-based automated scoring approaches to construct-irrelevant factors. This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system designed to score short essay-like open-response items in a situational judgment test. It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication. Duplicating large passages of text resulted in lower scores predicted by the system, on average, contradicting results from previous studies of non-LLM-based scoring systems, while off-topic responses were heavily penalized by the scoring system. These results provide encouraging support for the robustness of future LLM-based scoring systems when designed with construct relevance in mind.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
