We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.
Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values
Carnegie Mellon University(卡内基梅隆大学);Medical University of South Carolina(南卡罗来纳医科大学);Strategy Robot, Inc.(策略机器人公司);Strategic Machine, Inc.(战略机器公司);Optimized Markets, Inc.(优化市场公司)
Synergistic Weak-Strong Collaboration by Aligning Preferences
Microsoft Research(微软研究院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Nanyang Technological University(南洋理工大学)
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Hefei University of Technology(合肥工业大学);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);University of Toronto(多伦多大学)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。
Script-Strategy Aligned Generation: Aligning LLMs with Expert-Crafted Dialogue Scripts and Therapeutic Strategies for Psychotherapy
University of Amsterdam(阿姆斯特丹大学);National Institute of Informatics (NII)(日本信息处理研究所);Tilburg University(蒂尔堡大学);Southeast University(东南大学);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);Utrecht University(乌得勒支大学)
SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
Shanghai Science and Intelligence Institute, Shanghai, China(上海科学与智能研究所);Fudan University, Shanghai, China(复旦大学);Australian Institute for Machine Learning, The University of Adelaide(澳大利亚机器学习研究所,阿德莱德大学)
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
Forward KL Regularized Preference Optimization for Aligning Diffusion Policies
Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Tsinghua University(清华大学);Northwestern Polytechnical University Xi’an(西北工业大学(西安));Hong Kong University of Science and Technology(香港科技大学);ShenZhen Research Institute of Northwestern Polytechnical University(西北工业大学深圳研究院)
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University(萨尔兰大学);Independent Researcher(独立研究者);Bielefeld University(比勒菲尔德大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息研究所)