arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.01786cs.AIcs.LG
收藏

谁评判法官?LLM陪审团即需:构建可信的LLM评估系统

Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems

Wells Fargo Bank, N.A.(富国银行)

Xiaochuan Li, Ke Wang, Girija Gouda, Shubham Choudhary, Yaqun Wang, Linwei Hu, Joel Vaughan, Freddy Lecue

AI总结:

本文提出LLM陪审团即需,通过动态学习框架提升LLM评估的可靠性和可扩展性,实验表明其在关键决策中的相关性优于传统方法。

摘要:

随着大型语言模型(LLMs)被整合到高风险领域,对能够实时部署且在关键决策中可靠的方法需求日益增长。尽管人类评估可靠,但速度慢且成本高。单一LLM法官存在偏见,静态陪审团缺乏适应性。为克服这些限制,我们提出了LLM陪审团即需——一种动态、基于学习的框架,用于可扩展且情境感知的评估。我们的方法训练一组可靠性预测器,以评估LLM法官何时会与人类专家一致,利用标记分布、嵌入和结构性输入特征。这使能够实现完全自适应的评估,其中对于每个数据点,最优的最可靠法官陪审团会被动态选择,并使用其可靠性作为权重进行汇总。在摘要和RAG基准测试中,我们的动态陪审团系统在与人类判断的相关性方面显著高于单法官和静态陪审团基线。这些结果突显了自适应、基于学习的陪审团在构建可扩展、更可靠和可信的现代LLM评估系统方面的潜力,特别是在高风险领域。

英文摘要:

As Large Language Models (LLMs) become integrated into high-stakes domains, there is a growing need for evaluation methods that are both scalable for real-time deployment and reliable for critical decision-making. While human evaluation is reliable, it is slow and costly. Single LLM judges are biased, and static juries lack adaptability. To overcome these limitations, we propose LLM Jury-on-Demand - a dynamic, learning-based framework for scalable and context-aware evaluation. Our method trains a set of reliability predictors to assess when LLM judges will agree with human experts, leveraging token distributions, embeddings, and structural input features. This enables a fully adaptive evaluation where, for each data point, an optimal jury of the most reliable judges is dynamically selected, and their scores are aggregated using their reliability as weights. Experiments on summarization and RAG benchmarks show that our dynamic jury system achieves significantly higher correlation with human judgment than both single-judge and static-jury baselines. These results highlight the promise of adaptive, learning-based juries for building scalable, more reliable and trustworthy evaluation systems for modern LLMs in high-stakes domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑