弥合人类与大语言模型判断之间的鸿沟：理解和缩小差距
Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
AI总结：
Bridge通过统一统计框架弥合人类与LLM判断差距，改进评分并揭示系统性差异。
摘要：
大型语言模型越来越多地被用作裁判（LLM-as-a-judge）来对模型输出进行大规模评估，但它们的评估往往系统性地偏离人类判断。我们提出了Bridge，一个统一的统计框架，它在绝对评分和成对比较范式下明确地弥合人类和LLM评估之间的差距。Bridge为每个提示-响应对假定一个潜在的人类偏好分数，并将LLM的偏差建模为协变量的线性变换，这些协变量捕捉了差异源。这提供了一个简单且原则性的框架，用于改进LLM评分并表征人类和LLM之间的系统性差异。我们提供了一个高效的拟合算法，具有统计推断的渐近保证。使用六个LLM裁判和两个基准（BigGen Bench和Chatbot Arena），Bridge在准确性、校准和KL散度方面实现了与人类评分更高的一致性，并揭示了系统性的人与LLM差距。
英文摘要：
Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
