更公平的偏好引导出改进的人类对齐大型语言模型评判
Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
AI总结：
本文揭示LLM评估器的偏好脆弱性，并提出零样本提示优化框架ZEPO，通过优化偏好决策公平性，在无标注数据下显著提升了LLM评估器与人类评判的对齐度。
摘要：
大型语言模型（LLMs）作为评估语言生成质量的低成本且无需参考的评估器，展现出了有前景的能力。特别是，比较两段生成文本并确定偏好哪一个的成对LLM评估器，已在广泛的应用中被采用。然而，LLMs表现出偏好偏差，并且对提示设计具有令人担忧的敏感性。在这项工作中，我们首先揭示了LLMs的预测偏好可能是高度脆弱和倾斜的，即使在使用语义等价的指令时也是如此。我们发现，来自LLMs的更公平的预测偏好能够持续带来与人类更好对齐的评判。受这一现象启发，我们提出了一种自动的面向零样本评估的提示优化框架，ZEPO，旨在产生更公平的偏好决策并改善LLM评估器与人类评判的对齐。为此，我们提出了一种基于偏好决策公平性的零样本学习目标。ZEPO在代表性的元评估基准上，无需标注数据，即展现出相较于最先进LLM评估器的显著性能提升。我们的发现强调了偏好公平性与人类对齐之间的关键相关性，将ZEPO定位为弥合LLM评估器与人类评判之间差距的高效提示优化器。
英文摘要：
Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language generation quality. In particular, pairwise LLM evaluators, which compare two generated texts and determine the preferred one, have been employed in a wide range of applications. However, LLMs exhibit preference biases and worrying sensitivity to prompt designs. In this work, we first reveal that the predictive preference of LLMs can be highly brittle and skewed, even with semantically equivalent instructions. We find that fairer predictive preferences from LLMs consistently lead to judgments that are better aligned with humans. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO, which aims to produce fairer preference decisions and improve the alignment of LLM evaluators with human judgments. To this end, we propose a zero-shot learning objective based on the preference decision fairness. ZEPO demonstrates substantial performance improvements over state-of-the-art LLM evaluators, without requiring labeled data, on representative meta-evaluation benchmarks. Our findings underscore the critical correlation between preference fairness and human alignment, positioning ZEPO as an efficient prompt optimizer for bridging the gap between LLM evaluators and human judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
