arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.21140cs.LGcs.CLstat.APstat.ML
收藏

如何正确报告LLM作为评估者的评估结果

How to Correctly Report LLM-as-a-Judge Evaluations

University of Science and Technology of China(中国科学技术大学)

Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee

AI总结:

针对LLM作为评估者时存在偏差的问题,提出一种插件式校正框架,实现无偏估计和统计原理的不确定性量化,并证明在分布偏移下仍保持无偏性。

摘要:

大型语言模型(LLMs)被广泛用作模型响应的可扩展评估者,以替代人工标注者。然而,LLM评估者的不完美灵敏度和特异性会导致朴素评估分数产生偏差。我们提出一个简单的插件式框架,可校正此偏差并实现统计原理的不确定性量化。我们的框架构建置信区间,该区间同时考虑来自测试数据集和人工标注校准数据集的不确定性。此外,它采用自适应策略分配校准样本以获得更紧的区间。重要的是,我们刻画了由真实评估分数和LLM评估者的灵敏度与特异性定义的参数区间,在这些区间内,基于LLM的评估比仅人工评估产生更可靠的估计。此外,我们证明,与现有方法相比,我们的框架在测试集和校准集之间存在分布偏移时仍保持无偏性。

英文摘要:

Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑