小型语言模型知道它们犯错时吗?基于置信度的级联评分用于教育评估
Do Small Language Models Know When They're Wrong? Confidence-Based Cascade Scoring for Educational Assessment
AI总结:
研究通过置信度评估确定级联系统中何时升级任务,发现置信度差异影响评分准确性与成本,最佳级联在降低成本和延迟的同时接近大模型性能。
摘要:
大规模自动评分学生作品需要平衡准确性、成本和延迟。在
英文摘要:
Automated scoring of student work at scale requires balancing accuracy against cost and latency. In "cascade" systems, small language models (LMs) handle easier scoring tasks while escalating harder ones to larger LMs -- but the challenge is determining which cases to escalate. We explore verbalized confidence -- asking the LM to state a numerical confidence alongside its prediction -- as a routing signal. Using 2,100 expert-scored decisions from student-AI math conversations, we evaluate cascade systems built from GPT-5.4, Claude 4.5+, and Gemini 3.1 model pairs. We find that: (1) confidence discrimination varies widely across small LMs, with the best achieving AUROC 0.857 and the worst producing a near-degenerate confidence distribution; (2) confidence tracks human scoring difficulty, with lower LM confidence where annotators disagreed and took longer to score; (3) the best cascade approached large-LM accuracy (kappa 0.802 vs. 0.819) at 76% lower cost and 61% lower latency. Confidence discrimination is the bottleneck: the two small LMs with meaningful confidence variance yielded cascades with no statistically detectable kappa loss, while the third -- whose confidence was near-degenerate -- could not close the accuracy gap regardless of threshold. Small LMs with strong discrimination let practitioners trade cost for accuracy along the frontier; those without it do not.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。