Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.
Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy
Nankai University(南开大学);Tsinghua University(清华大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Wuhan University(武汉大学)
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用关键实验室);Ministry of Education(教育部)
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学计算机科学系);Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学电气与计算机工程系)
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University(国立台湾大学);Texas A&M University(德克萨斯A&M大学);Appier AI Research(Appier人工智能研究院);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所)
FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Communication Content Cognition(通信内容认知国家重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学)
Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(人工智能向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Kling Team, Kuaishou Technology(快手科技 Kling 团队);NSERC CRC Chair(加拿大NSERC CRC主席)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Tencent Inc.(腾讯公司)
Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity
School of Artificial Intelligence, OPtics and ElectroNics, Northwestern Polytechnical University(人工智能学院、光学与电子学院、西北工业大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信);School of Economics and Management, Guangxi Normal University(经济管理学院,广西师范大学)
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
Graduate School of AI, POSTECH, Pohang, Republic of Korea(POSTECH人工智能研究生院);Department of CSE, POSTECH, Pohang, Republic of Korea(POSTECH计算机科学与工程系);National AI Research Lab, Seoul, Republic of Korea(首尔国家人工智能研究实验室)