基于通用幂均值的温度控制判决聚合中的AI系统评估适应性
Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean
AI总结:
本文提出TCVA方法,结合五级判决评分系统和通用幂均值聚合,通过温度参数控制评估严格性,实验显示其在忠实度上与RAGAS相当,且优于DeepEval。
摘要:
现有LLM基于AI系统评估方法如LLM-as-a-Judge、判决系统和NLI无法始终与人类评估一致,因为它们不能适应应用领域。本文提出了温度控制判决聚合(TCVA)方法,结合五级判决评分系统和通用幂均值聚合,以及直观的温度参数T[0.1, 1.0]来控制评估严格性。低温产生悲观评分,适合安全关键领域;高温产生宽松评分,适合对话AI。在三个基准数据集上进行实验评估,使用人类Likert尺度标注(SummEval和USR)显示,TCVA在忠实度上与RAGAS相当(Spearman=0.667 vs. 0.676),同时在一致性上优于DeepEval。该方法在调整温度参数时不需要额外的LLM调用。
英文摘要:
Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain. This paper presents Temperature-Controlled Verdict Aggregation (TCVA), a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor. Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI. Experimental evaluation on three benchmark datasets with human Likert-scale annotations (SummEval and USR) shows that TCVA achieves correlation with human judgments comparable to RAGAS on faithfulness (Spearman = 0.667 vs. 0.676) while consistently outperforming DeepEval. The method requires no additional LLM calls when adjusting the temperature parameter.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。