平衡准确率：评估大语言模型判官的正确指标——通过Youden的J统计量解释
Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic
AI总结：
本文通过Youden的J统计量证明平衡准确率是评估LLM判官的更优指标，通过分析和实验展示其在选择判官时提升分类器鲁棒性的效果。
摘要：
严格评估大语言模型（LLM）依赖于通过 desirable 或 undesirable 行为的普遍性来比较模型，如任务通过率或政策违规。这些普遍性估计由分类器生成，无论是LLM-as-a-judge还是人类标注者，因此选择分类器对可信评估至关重要。常用的指标如准确率、精确率和F1值对类别不平衡和任意正类选择敏感，可能偏袒扭曲普遍性估计的判官。我们展示Youden的J统计量在理论上与选择最佳判官比较模型一致，而平衡准确率是J的等价线性变换。通过分析论证和实证例子及模拟，我们展示了使用平衡准确率选择判官如何导致更优、更稳健的分类器选择。
英文摘要：
Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
