DPO is an effective preference optimization algorithm. However, the DPO-tuned models tend to overfit on the dispreferred samples, manifested as overly long generations lacking diversity. While recent regularization approaches have endeavored to alleviate this issue by modifying the objective function, they achieved that at the cost of alignment performance degradation. In this paper, we innovatively incorporate regularization from the perspective of weight updating to curb alignment overfitting. Through the pilot experiment, we discovered that there exists a positive correlation between overfitting and the hyperspherical energy fluctuation. Hence, we introduce orthogonal finetuning for DPO via a weight-Rotated Preference Optimization (RoPO) method, which merely conducts rotational and magnitude-stretching updates on the weight parameters to maintain the hyperspherical energy invariant, thereby preserving the knowledge encoded in the angle between neurons. Extensive experiments demonstrate that our model aligns perfectly with human preferences while retaining the original expressive capacity using only 0.0086% of the trainable parameters, suggesting an effective regularization against overfitting. Specifically, RoPO outperforms DPO by up to 10 points on MT-Bench and by up to 2.8 points on AlpacaEval 2, while enhancing the generation diversity by an average of 6 points.
Align Generative Artificial Intelligence with Human Preferences: A Novel Large Language Model Fine-Tuning Method for Online Review Management
Department of Information Systems and Operations Management, The University of Texas at Arlington(信息系统与运营管理系,德克萨斯大学阿灵顿分校);Department of Management Information Systems, University of Arizona(管理信息学系,亚利桑那大学)
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);University of Tübingen(蒂宾根大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);The Alan Turing Institute(艾伦·图灵研究所)
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);The Chinese University of Hong Kong(香港中文大学);University of Cambridge(剑桥大学);The Alan Turing Institute(艾伦·图灵研究所)
CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Xiaohongshu Inc(小红书公司)
Training Language Models with Language Feedback at Scale
New York University(纽约大学);FAR AI;University of the Basque Country UPV/EHU(巴斯克大学);University of Sussex(萨塞克斯大学);Genentech(基因泰克公司);CIFAR LMB(加拿大高级研究院分子生物学与遗传学实验室);Anthropic(Anthropic公司)
OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);RUC(中国人民大学);USTC(University of Science and Technology of China);NTU(National University of Technology);NUS(National University of Singapore);USC(University of Southern California);SCU(Sichuan University);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation
Institute for Logic, Language and Computation, University of Amsterdam(逻辑、语言与计算研究所,阿姆斯特丹大学);Center for Language and Cognition, University of Groningen(语言与认知中心,格罗宁根大学)
Reward Models Inherit Value Biases from Pretraining
Department of Experimental Psychology, University of Oxford(心理学系,牛津大学);Department of Computer Science, University of Oxford(计算机科学系,牛津大学);AI/ML Research Group, Universitat Pompeu Fabra(人工智能/机器学习研究组,庞培法华大学);Oxford Internet Institute, University of Oxford(牛津互联网研究所,牛津大学)
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Network Analysis and Social Influence Modelling (NASIM) Lab(网络分析与社会影响建模实验室);School of Physics Maths and Computing, The University of Western Australia(西澳大学物理数学与计算学院);School of Psychological Science, The University of Western Australia(西澳大学心理科学学院);School of Computing, Macquarie University(麦考瑞大学计算机学院)
Weights-Rotated Preference Optimization for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)
OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(复杂系统认知与决策智能重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
SAND: Boosting LLM Agents with Self-Taught Action Deliberation
University of California San Diego(加利福尼亚大学圣地亚哥分校);Adobe Research(Adobe研究院);University of New South Wales(新南威尔士大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61)
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Manning College of Information and Computer Sciences(Manning信息与计算机科学学院);Ensemble HP;Columbia University(哥伦比亚大学);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校)