Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMs align with human values. An essential part of ensuring that LLMs are aligned for all people is accounting for a diverse set of values. This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity in models trained with direct preference optimization. Our framework rigorously assesses how well models generalize after a finite number of gradient steps, reflecting real-world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, we provide a bound on the generalization error that demonstrates the challenges of effectively learning a wide set of concepts or values. These insights are empirically validated on contemporary LLMs, underscoring the practical relevance of our theory.
University of Sydney(悉尼大学);Imperial College London(伦敦帝国理工学院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Virginia Tech(弗吉尼亚理工大学);University of Texas at Dallas(德克萨斯大学达拉斯分校)
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学计算机科学与工程保罗·G·艾伦学校);Max Planck Institute for Intelligent Systems, Tübingen, Germany(德国图宾根马克斯·普朗克智能系统研究所);Jinesis Lab, University of Toronto & Vector Institute(多伦多大学Jinesis实验室及向量研究所)
GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO
College of Computing and Data Science,Nanyang Technological University (NTU), Singapore(南洋理工大学计算与数据科学学院);Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly(LILY),NTU,Singapore(南洋理工大学与UBC联合卓越老年活跃生活研究中心(LILY))
Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
Department of Applied Mathematics and Statistics(应用数学与统计学系);Stony Brook University(石溪大学);Institute for Advanced Computational Science(先进计算科学研究所);Center of Excellence in Wireless and Information Technology (CEWIT)(无线与信息技术卓越中心);AI Innovation Institute(人工智能创新研究院);Bloomberg(彭博);Toronto, ON M5J 2S1(多伦多,ON M5J 2S1);San Francisco, CA 94105(旧金山,CA 94105);Bloomberg New York, NY 10022(纽约,NY 10022)
Less is More: Improving LLM Alignment via Preference Data Selection
University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Massachusetts Institute of Technology(麻省理工学院);Alibaba Cloud Computing(阿里云计算);MoE Key Lab of BIPC, University of Science and Technology of China(中国科学技术大学MoE关键实验室)
Structured Scaling of AI Discovery Across Diverse Scientific Domains
Wizard Intelligence Learning Lab, Stanford University(智能巫师学习实验室,斯坦福大学);Peking University(北京大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
Bellini College of AI, Cybersecurity, and Computing(人工智能、网络安全与计算学院);University of South Florida, Tampa, Florida, USA(佛罗里达州塔帕斯大学);University of California, Davis, California, USA(加州大学戴维斯分校)