CheckEval：一种使用清单评估文本生成的可靠 LLM-as-a-Judge 框架
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists
AI总结：
针对现有LLM-as-a-Judge方法评分不一致的问题，提出基于清单的评估框架CheckEval，通过分解二值问题提升评分可靠性，实验证明其与人类判断高度相关，且显著提升了评估模型间的一致性并降低方差。
摘要：
现有的评估文本生成的 LLM-as-a-Judge 方法存在评分不一致问题，不同评估模型间一致性低且评分方差高。我们将其归因于现有协议中主观的评估标准与 Likert scale 打分方式。为解决此问题，我们提出了 CheckEval，这是一种基于清单的评估框架，通过分解的二值问题提升评分可靠性。通过对多个数据集上的 12 个评估模型进行实验，我们首先证明 CheckEval 与人类判断高度相关。更重要的是，CheckEval 将评估模型间的平均一致性大幅提升了 0.45，并降低了分数方差。此外，CheckEval 分数更具可解释性，因为它将评估标准分解为可追溯的二值决策，从而允许分析驱动质量判断的具体属性。
英文摘要：
Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models. We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols. To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments. More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance. CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
