偏好调优依赖于高质量的偏好数据，而收集这些数据通常既昂贵又耗时。在本文中，我们提出了Dr.SoW（Density Ratio of Strong over Weak，强弱密度比），这是一种高性价比的方法，通过利用现成的LLM进行偏好数据标注，消除了对人工标注的依赖。Dr.SoW使用对齐较好的LLM与对齐较差的LLM之间的对数密度比作为奖励信号。我们在221对不同的LLM上评估了Dr.SoW，并实证发现配对模型的性能差距与奖励信号的质量之间存在强相关性。这一见解为选择用于数据标注的LLM提供了实用指南。此外，我们引入了一个端到端流水线，可根据用户查询领域定制奖励函数。无需微调，它即可提高特定领域评估的准确率。使用一对Mistral-7B模型，Dr.SoW获得了82.6的RewardBench分数，超越了同类模型中训练效果最佳的奖励函数，并在Safety（91.0）和Reasoning（88.0）领域展现出与SoTA模型相当的竞争性能。进一步，我们使用Dr.SoW标注的数据对Llama-3-8B-Instruct进行偏好调优。我们的方法推动Llama-3-8B在ArenaHard上达到37.4%（+15.1%）的胜率，在长度控制的AlpacaEval 2.0上达到40.7%（+17.8%）的胜率。
英文摘要：
Preference tuning relies on high-quality human preference data, which is often expensive and time-consuming to gather. In this paper, we introduce Dr.SoW (Density Ratio of Strong over Weak) a cost-effective method that eliminates the reliance for human annotation by leveraging off-the-shelf LLMs for preference data annotation. Dr.SoW uses the log-density ratio between a better-aligned and a less-aligned LLM as a reward signal. We evaluate Dr.SoW across 221 different LLM pairs and empirically find a strong correlation between the performance gap of the paired models and the quality of the reward signal. This insight provides a practical guideline for selecting LLMs for data annotation.
Additionally, we introduce an end-to-end pipeline that customizes reward functions based on user query domains. Without fine-tuning, it improves accuracy on domain-specific evaluations. With a pair of Mistral-7B models, Dr.SoW achieves a RewardBench score of 82.6, outperforming the best trained reward functions from same model class and demonstrating competitive performance against SoTA models in Safety (91.0) and Reasoning (88.0) domains. Further, we preference-tune Llama-3-8B-Instruct using data annotated by Dr.SoW. Our approach pushes Llama-3-8B to achieve a 37.4 % (+15.1 %) win rate on ArenaHard and a 40.7 % (+17.8 %) win rate on length-controlled AlpacaEval 2.0.
Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies(PKU), MOE, China（教育部高可信软件技术重点实验室（北京大学））；Beijing Jiaotong University（北京交通大学）
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai AI Lab（上海人工智能实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
MaiNLP；Center for Information and Language Processing（信息与语言处理中心）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Vienna（维也纳大学）；LTL；University of Cambridge（剑桥大学）
Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
Australian Artificial Intelligence Institute (AAII), University of Technology Sydney, Australia（澳大利亚人工智能研究所（AAII）、悉尼技术大学）；Centre for Frontier AI Research, Institute of High Performance Computing（前沿人工智能研究中心、高性能计算研究所）；Agency for Science, Technology and Research, Singapore（科技研究局（新加坡））；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院、南洋理工大学）
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
Department of Computer Science (DIKU), University of Copenhagen（哥本哈根大学计算机科学系（DIKU））；Centre for Language Technology (CST), University of Copenhagen（哥本哈根大学语言技术中心）；Pioneer Centre for Artificial Intelligence（先锋人工智能中心）
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University（萨尔兰大学）；Independent Researcher（独立研究者）；Bielefeld University（比勒菲尔德大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
Random Direct Preference Optimization for Radiography Report Generation
Artificial Intelligence Research Institute (AIRI)（人工智能研究院）；Skolkovo Institute of Science and Technology（斯克罗夫学院）；Institute for Information Transmission Problems（信息传输问题研究所）
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Department of Computer Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机科学与数据科学学院）
