Towards a Unified View of Preference Learning for Large Language Models: A Survey
Peking University(北京大学);Alibaba Group(阿里巴巴集团);Shanghai Jiao Tong University(上海交通大学);Zhongguancun Laboratory(中关村实验室);Microsoft(微软);University of Waterloo(滑铁卢大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences.
A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
University of California, San Diego(加利福尼亚大学圣地亚哥分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Adobe Research(奥多比研究院);The Ohio State University(俄亥俄州立大学);Allen Institute for AI(艾伦人工智能研究所)
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
Ant Group(蚂蚁集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
AI Alignment through a Game-theoretic Lens: A Survey
James Cook University(詹姆斯库克大学);Western Sydney University(西悉尼大学);Adelaide University(阿德莱德大学);CSIRO(联邦科学与工业研究组织);The University of Osaka(大阪大学);Nanjing University of Science and Technology(南京理工大学);Macquarie University(麦考瑞大学);La Trobe University(拉筹伯大学)
Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy
School of Computing, Macquarie University(麦考瑞大学计算机学院);Department of Computer Engineering and Information Technology, Shiraz University of Technology(谢里兹技术大学计算机工程与信息技术系)
Advances and Challenges in Conversational Recommender Systems: A Survey
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);University of Amsterdam(阿姆斯特丹大学);Ahold Delhaize(阿霍德德尔海兹)