Spread Preference Annotation：用于高效LLM对齐的直接偏好判断
Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
AI总结：
提出SPA框架，利用少量人类标注数据和LLM的logits提取偏好标签进行迭代自标注，并结合噪声感知算法，仅用3.3%真实标签即实现了高效的LLM对齐。
摘要：
将大型语言模型（LLMs）与人类偏好对齐已成为获得最先进性能的关键组成部分，但构建大型人类标注的偏好数据集会带来巨大成本。为解决此问题，我们提出了一种新框架，即带直接偏好判断的Spread Preference Annotation（SPA），该框架仅使用极少量的标注偏好数据即可提升LLMs的对齐效果。我们的核心思想是利用小规模（种子）数据中的人类先验知识，通过迭代生成响应并利用自标注的偏好数据从中学习，逐步提升LLM的对齐能力。具体而言，我们提出从LLM的logits中推导偏好标签，以显式提取模型固有的偏好。与以往使用外部奖励模型或隐式上下文学习的方法相比，我们观察到所提出的方法显著更有效。此外，我们引入了一种噪声感知偏好学习算法，以降低生成偏好数据中低质量带来的风险。我们的实验结果表明，所提出的框架显著提升了LLMs的对齐效果。例如，在AlpacaEval 2.0上，与使用Ultrafeedback数据中全部真实偏好标签或最先进基线的情况相比，我们仅使用3.3%的真实偏好标签便取得了更优的对齐性能。
英文摘要：
Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
