Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories.
We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Wroclaw University of Science and Technology (WUST)(沃拉布大学科学与技术学院);National Research Institute (NASK)(国家研究 institute);Institute of Computer Science, Polish Academy of Sciences (IPI PAN)(波兰科学院计算机科学研究所)
D2PO: Discriminator-Guided DPO with Response Evaluation Models
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Princeton University(普林斯顿大学)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Microsoft Research AI4Science(微软研究院AI4Science);University College London(伦敦大学学院)
Fudan University(复旦大学);Princeton University(普林斯顿大学);University of California Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);Google DeepMind(谷歌DeepMind);University of Virginia(弗吉尼亚大学)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)
The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
University College Dublin(都柏林大学学院);CeADAR(CeADAR(爱尔兰人工智能中心))
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
Georgia Institute of Technology(佐治亚理工学院);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学)