透视偏好：揭示用于对齐大语言模型的反馈获取机制
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
AI总结：
本研究分析了大语言模型对齐中评分与排序两种稀疏反馈协议，发现两者偏好不一致率达60%，且影响模型评估，揭示评估方法对反馈协议的依赖。
摘要：
将大语言模型（LLMs）与人类价值观和意图对齐，关键涉及使用人类或AI反馈。虽然密集的反馈标注获取和整合成本高昂，但稀疏反馈在结构设计上存在选择，即在评分（例如，在1-7的尺度上对响应A打分）和排序（例如，响应A是否优于响应B？）之间进行选择。在这项工作中，我们分析了这一设计选择对LLMs对齐和评估的影响。我们发现了一个不一致性问题，即从评分和排序中推断出的偏好在人类和AI标注者中均有60%的显著分歧。我们随后的分析识别了标注者偏差的多个方面来解释这一现象，例如人类标注者会给予更密集的响应更高评分，而在成对判断时更偏好准确性。令我们惊讶的是，我们还观察到反馈协议的选择也对对齐后LLMs的评估有显著影响。具体来说，我们发现利用排序数据进行对齐的LLMs（例如模型X）在使用基于排序的评估协议（即X/Y的响应是否优于参考响应？）时，比利用评分数据进行对齐的模型（例如模型Y）更受青睐，但在基于评分的评估协议（即在1-7的尺度上对X/Y的响应进行评分）下则不然。我们的发现因此揭示了评估语言模型实际效用方法中的关键缺陷，以及这些方法对用于对齐的反馈协议的强烈依赖性。我们的代码和数据可在https://github.com/Hritikbansal/sparse_feedback获取。
英文摘要：
Aligning large language models (LLMs) with human values and intents critically involves the use of human or AI feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural design choice between ratings (e.g., score Response A on a scale of 1-7) and rankings (e.g., is Response A better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMs. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena, such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments. To our surprise, we also observe that the choice of feedback protocol also has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?) but not with a rating-based evaluation protocol (score Rank X/Y's response on a scale of 1-7). Our findings thus shed light on critical gaps in methods for evaluating the real-world utility of language models and their strong dependence on the feedback protocol used for alignment. Our code and data are available at https://github.com/Hritikbansal/sparse_feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
