Iterative data generation and model re-training can effectively align large language models(LLMs) to human preferences. The process of data sampling is crucial, as it significantly influences the success of policy improvement. Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs. In this work, we propose a more effective sampling method, named Preference-Guided Reflective Sampling (PRS). Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. It leverages adaptive self-refinement techniques to better explore the sampling space. By specifying user preferences in natural language, PRS can further optimize response generation according to these preferences. As a result, PRS can align models to diverse user preferences. Our experiments demonstrate that PRS generates higher-quality responses with significantly higher rewards. On AlpacaEval and Arena-Hard, PRS substantially outperforms repeated random sampling in best-of-$N$ sampling. Moreover, PRS shows strong performance when applied in iterative offline RL training.
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）；Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系， 威特沃特斯兰大学）
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation（微软公司）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Southern California（南加州大学）
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Evaluating and Aligning CodeLLMs on Human Preference
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University（北京大学计算机科学系）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；National Research and Engineering Center of Software Engineering, Peking University（北京大学软件工程研究中心）
Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Model
Department of Gastroenterology, Renmin Hospital, Wuhan University（武汉大学人民医院消化内科）；School of Computer Science, Wuhan University（武汉大学计算机学院）；State Grid Corporation（国家电网公司）
Panacea: Pareto Alignment via Preference Adaptation for LLMs
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；BIGAI（北京智源人工智能研究院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Yuanpei College, Peking University（北京大学元培学院）
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）
提出自参考 AI 反馈框架，让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据，显著提升强化学习对齐效果。
ERA-IT: Aligning Semantic Models with Revealed Economic Preference for Real-Time and Explainable Patent Valuation
School of Computing（计算学院）；Macquarie University（麦考瑞大学）；School of Computer Science（计算机科学学院）；University of Technology Sydney（悉尼技术大学）；School of Management（管理学院）；Zhejiang University（浙江大学）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
