arXiv 2606.051802026-04-18cs.CLcs.AI

从评分到解释：评估基于量规的教学质量评估中的SHAP和LLM理由

From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment

Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Lund University（吕勒奥大学）；University of Tübingen（图宾根大学）；Stanford Graduate School of Education（斯坦福大学教育研究生院）；Harvard Graduate School of Education（哈佛大学教育研究生院）

Ivo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer, Ulrich Trautwein, Dorottya Demszky, Heather Hill, Enkelejda Kasneci

备注：: Accepted to Findings of ACL 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一个结合SHAP和LLM理由的框架，用于基于量规的评分模型的可解释性，并在课堂转录数据上评估其忠实性和可迁移性。

摘要：

自动化评分模型越来越多地被用于为复杂的语言表现（包括课堂转录）分配基于量规的质量评级，但它们通常很少提供关于为什么产生特定分数的见解。我们提出了一个通用的框架，用于基于量规的评分的句子级可解释性，该框架将模型无关的Shapley值归因与大型语言模型（LLM）生成的理由相结合。在使用NCTE语料库的CLASS框架的反馈质量维度上实例化，该框架能够系统地比较微调的预训练语言模型（PLM）和提示的LLM在评分性能和解释忠实性方面的表现。在6k个带注释的转录片段中，微调的PLM在预测准确性上优于LLM，但表现出向中等尺度分数的标签压缩。基于删除的测试表明，SHAP识别出可靠驱动模型预测的句子，产生的预测变化通常比LLM生成的理由更大且更连贯。跨模型分析进一步揭示，SHAP归因在不同架构间稳健地迁移，而LLM理由的影响有限且不一致。总体而言，研究结果表明，SHAP为基于量规的评分提供了更忠实和可迁移的解释，并且所提出的框架为在高风险教育环境和其他基于量规的语言评估任务中评估评分模型及其解释提供了原则性基础。

英文摘要：

Automated scoring models are increasingly used to assign rubric-based quality ratings to complex language performances, including classroom transcripts, yet they typically provide little insight into why a particular score is produced. We propose a general framework for sentence-level interpretability of rubric-based scoring that combines model-agnostic Shapley-value attributions with rationales generated by large language models (LLMs). Instantiated on the Quality of Feedback dimension of the CLASS framework using the NCTE corpus, the framework enables systematic comparison of fine-tuned pretrained language models (PLMs) and prompted LLMs on both scoring performance and explanation faithfulness. Across 6k annotated transcript segments, fine-tuned PLMs outperform LLMs in prediction accuracy but exhibit label compression toward mid-scale scores. Deletion-based tests show that SHAP identifies sentences that reliably drive model predictions, producing typically larger and more coherent prediction shifts than LLM-generated rationales. Cross-model analyses further reveal that SHAP attributions transfer robustly across architectures, whereas LLM rationales exert limited and inconsistent influence. Overall, the findings demonstrate that SHAP provides more faithful and transferable explanations for rubric-based scoring, and that the proposed framework offers a principled basis for evaluating both scoring models and their explanations in high-stakes educational settings and other rubric-based language assessment tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。