我们该信任谁？LLM作为陪审团进行比较评估
Who can we trust? LLM-as-a-jury for Comparative Assessment
AI总结：
针对LLM作为评估者时判断不一致和可靠性差异的问题，提出BT-sigma模型，通过引入判别参数联合推断项目排名和法官可靠性，优于平均聚合方法。
摘要：
大型语言模型（LLMs）越来越多地被用作自动评估器，用于自然语言生成评估，通常采用成对比较判断。现有方法通常依赖单一法官或聚合多个法官并假设其可靠性相同。在实践中，LLM法官在不同任务和评估方面的表现差异很大，其判断概率可能存在偏差和不一致。此外，用于法官校准的人工标注监督可能不可用。我们首先通过实验证明LLM比较概率的不一致性存在，并表明这限制了直接基于概率排名的有效性。为解决此问题，我们研究了LLM作为陪审团的设置，并提出了BT-sigma，这是Bradley-Terry模型的一种法官感知扩展，为每个法官引入一个判别参数，仅从成对比较中联合推断项目排名和法官可靠性。在基准NLG评估数据集上的实验表明，BT-sigma始终优于基于平均的聚合方法，并且学习到的判别参数与LLM判断的循环一致性的独立度量高度相关。进一步分析揭示，BT-sigma可以解释为一种无监督校准机制，通过建模法官可靠性来改进聚合。
英文摘要：
Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
