arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04820cs.CLcs.CY
收藏

自评分的反高潮:对AI在简答短处和用词弱点的元分析理解

Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses

Stanford University(斯坦福大学)

Michael Hardy

AI总结:

本文通过元分析研究AI在简答评分中的不足,发现人类评分难度对LLM性能无显著影响,且某些人类易评任务对LLM更难,同时探讨了LLM技术对评分的影响因素。

摘要:

自动化简答评分滞后于其他LLM应用。我们对890项系统综述中的LLM简答评分研究进行了元分析,利用混合效应元回归模型对传统效应量Quadratic Weighted Kappa(QWK)进行建模。我们定量表明,人类专家评分儿童书面作业的难度水平对LLM性能无显著统计影响。特别地,我们发现某些由人类评分者认为最简单的评分任务却是LLM最困难的。无论是由于研究人员的拙劣实现还是可追溯到自回归训练的模式,解码器-only架构在与编码器相比时平均表现差0.37-这在与人类一致性的差异上是显著的。此外,我们测量了各种LLM技术对成功评分的贡献,如分词器词汇量,其表现出递减回报——可能由于未训练的token。研究结果支持设计更能够预见自回归模型已知统计缺陷的系统。最后,我们提供了额外实验以说明在高风险教育环境中,LLM在用词和分词敏感性和偏见揭示方面的问题,其中LLM显示出种族歧视。本研究的代码和数据可供使用。

英文摘要:

Automated short-answer scoring lags other LLM applications. We meta-analyze 890 culminating results across a systematic review of LLM short-answer scoring studies, modeling the traditional effect size of Quadratic Weighted Kappa (QWK) with mixed effects metaregression. We quantitatively illustrate that that the level of difficulty for human experts to perform the task of scoring written work of children has no observed statistical effect on LLM performance. Particularly, we show that some scoring tasks measured as the easiest by human scorers were the hardest for LLMs. Whether by poor implementation by thoughtful researchers or patterns traceable to autoregressive training, on average decoder-only architectures underperform encoders by 0.37--a substantial difference in agreement with humans. Additionally, we measure the contributions of various aspects of LLM technology on successful scoring such as tokenizer vocabulary size, which exhibits diminishing returns--potentially due to undertrained tokens. Findings argue for systems design which better anticipates known statistical shortcomings of autoregressive models. Finally, we provide additional experiments to illustrate wording and tokenization sensitivity and bias elicitation in high-stakes education contexts, where LLMs demonstrate racial discrimination. Code and data for this study are available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑