Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and a 74.6% increase in story diversity, while maintaining similar win rates as standard baselines. On general instruction following, DivPO results in a 46.2% increase in diversity, and a 2.4% winrate improvement compared to DPO.
MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海智能信息处理重点实验室)
Multi-Objective Exploration and Preference Optimization via Mutual Information
School of Computer, Beihang University(北京航空航天大学计算机学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰AGI实验室,中国电信人工智能技术(北京)有限公司)
Multi-Response Preference Optimization with Augmented Ranking Dataset
INMED DATA;Yonsei University(延世大学);Asan Medical Center(峨山医院);University of Ulsan College of Medicine(蔚山大学医学院);Asan Institute for Life Sciences(峨山生命科学研究所)
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
AI Technology Center of OVB, Tencent(腾讯OVB人工智能技术中心);Platform and Content Group, Tencent(腾讯平台与内容事业群);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning
Northwestern Polytechnical University, School of Automation, Xi’an, China(西北工业大学自动化学院,西安,中国);Northwestern Polytechnical University, School of Cybersecurity, Xi’an, China(西北工业大学网络安全学院,西安,中国);Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou, China(浙大宁波理工学院滨海人工智能研究院,杭州,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国)
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Department of Statistics, Rutgers University, New Brunswick, United States;Department of Computer Science, Rutgers University, New Brunswick, United States;College of Management of Technology, EPFL, Switzerland;Department of Computer Science, ETH Zurich, Switzerland
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)