Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Wroclaw University of Science and Technology (WUST)(沃拉布大学科学与技术学院);National Research Institute (NASK)(国家研究 institute);Institute of Computer Science, Polish Academy of Sciences (IPI PAN)(波兰科学院计算机科学研究所)
One Model for All: Multi-Objective Controllable Language Models
Ruhr University Bochum(波鸿鲁尔大学);Eindhoven University of Technology(埃因霍温理工大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Liverpool(利物浦大学)
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Department of Statistics, Rutgers University, New Brunswick, United States;Department of Computer Science, Rutgers University, New Brunswick, United States;College of Management of Technology, EPFL, Switzerland;Department of Computer Science, ETH Zurich, Switzerland
One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity
School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(哈佛大学自然与人工智能研究学院);Department of Psychology, Harvard University(哈佛大学心理学系)
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University(萨尔兰大学);Independent Researcher(独立研究者);Bielefeld University(比勒菲尔德大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息研究所)
Towards Comprehensive Preference Data Collection for Reward Modeling
Renmin University of China(中国人民大学);Gaoling School of Artificial Intelligence(高瓴人工智能学院);Kuaishou Technology(快手科技);University of Chinese Academy of Sciences(中国科学院大学)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Microsoft Research AI4Science(微软研究院AI4Science);University College London(伦敦大学学院)
Adversarial Preference Learning for Robust LLM Alignment
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)技术有限公司);University of Auckland(奥克兰大学);China Telecom Corporation Limited Beijing Research Institute(中国电信股份有限公司北京研究院)