RIFT:一种基于Rubric的故障模式分类和自动诊断
The Hitchhikers Guide to Rubric Quality Understanding and Enrichment
AI总结:
本文提出RIFT框架,通过系统化分类rubric的故障模式,提升评估的可靠性与有效性,通过人类标注和自动指标验证了其一致性与实用性。
摘要:
基于Rubric的评估广泛应用于LLM基准测试和训练流程,用于开放性、不易验证的任务。尽管先前工作已证明通过下游信号如强化学习结果来展示rubric的有效性,但尚无系统方法从这些聚合或下游信号中诊断rubric本身的故障。为解决这一差距,我们引入RIFT:RubrIc Failure mode Taxonomy,一种系统化表征rubric组成和设计中故障模式的分类体系。RIFT包含八种故障模式,分为三个高级类别:可靠性故障、内容有效性故障和后果有效性故障。RIFT通过扎根理论方法,通过迭代标注来自五个不同数据源的rubric,直到不再发现新的故障模式。我们通过测量独立人类标注者之间的共识来评估分类的一致性,观察到总体共识良好(87%配对共识和0.64平均Cohen's kappa)。最后,为支持可扩展诊断,我们提出自动rubric质量指标,并展示这些指标与人类故障模式标注一致,达到最高0.925 F1。
英文摘要:
Rubrics distill notions of expert quality and measure agent performance. However, the quality of rubrics themselves have not been systematically measured and are often left to downstream performance. We import apparatuses from measurement theory built for exactly this: quantitative signals based on the rubric's content, and introduce the RubrIc-Failure Taxonomy (RIFT), of nine possible ways a rubric fails, organized under reliability and content validity. Every mode leaves a distinct signature. To show the signals track failure causally, we seed 720 corruptions, injecting each RIFT mode into clean rubrics at known severity levels. A linear probe over the signals identifies which mode was injected at $75.0\%$ accuracy, beating $56.7\%$ for a frontier model asked to name the failure directly. Surprisingly across GDPval and Terminal-Bench, 10 of 48 expert-authored rubrics weight their criteria backwards, putting more of the score on requirements an expert panel judged less essential. This means a response can fail what matters most and still be graded well. This paper serves as a comprehensive guide on how to understand failure modes in rubrics and create better versions using quality signals, causal experiments, and provides a taxonomy with its rules and examples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。