Compare without Despair: 基于生成可分性的可靠偏好评估
Compare without Despair: Reliable Preference Evaluation with Generation Separability
AI总结：
针对成对偏好评估中因模型生成相似或随机解码差异导致的评分不一致问题，提出可分性元评估指标，验证其能提升评分一致性，还可用于基准价值判断与ELO排序优化，助力LLM可靠偏好评估。
摘要：
通过成对偏好判断对生成语言进行人工评估的方式十分普遍。然而在常见场景下，例如某模型对的生成结果高度相似，或是随机解码导致生成结果差异较大时，会出现偏好评分不一致的问题。我们通过提出一种元评估指标——可分性（separability）来应对这些挑战，该指标用于评估测试样本适合成对偏好评估的程度。对于一个候选测试样本，可分性会从一对模型中采样多个生成结果，并衡量两组生成结果的可区分程度。我们的实验表明，可分性值较高的样本能从人工评分者和自动评分者处得到更一致的偏好评分。此外，可分性的分布有助于了解哪些测试基准对模型比较更有价值。最后，我们将可分性融入ELO评分体系，考量每个测试样本对大语言模型（LLM）可靠排序的适配性。总体而言，可分性对实现LLM在人工与自动评分下一致、高效且鲁棒的偏好评估具有重要意义。
英文摘要：
Human evaluation of generated language through pairwise preference judgments is pervasive. However, under common scenarios, such as when generations from a model pair are very similar, or when stochastic decoding results in large variations in generations, it results in inconsistent preference ratings. We address these challenges by introducing a meta-evaluation measure, separability, which estimates how suitable a test instance is for pairwise preference evaluation. For a candidate test instance, separability samples multiple generations from a pair of models, and measures how distinguishable the two sets of generations are. Our experiments show that instances with high separability values yield more consistent preference ratings from both human- and auto-raters. Further, the distribution of separability allows insights into which test benchmarks are more valuable for comparing models. Finally, we incorporate separability into ELO ratings, accounting for how suitable each test instance might be for reliably ranking LLMs. Overall, separability has implications for consistent, efficient and robust preference evaluation of LLMs with both human- and auto-raters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
