Learning from preference feedback is essential for aligning large language models (LLMs) with human values and improving the quality of generated responses. However, existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale. In this work, we present PUGC, a novel framework that leverages implicit human Preferences in unlabeled User-Generated Content (UGC) to generate preference data. Although UGC is not explicitly created to guide LLMs in generating human-preferred responses, it often reflects valuable insights and implicit preferences from its creators that has the potential to address readers' questions. PUGC transforms UGC into user queries and generates responses from the policy model. The UGC is then leveraged as a reference text for response scoring, aligning the model with these implicit preferences. This approach improves the quality of preference data while enabling scalable, domain-specific alignment. Experimental results on Alpaca Eval 2 show that models trained with DPO and PUGC achieve a 9.37% performance improvement over traditional methods, setting a 35.93% state-of-the-art length-controlled win rate using Mistral-7B-Instruct. Further studies highlight gains in reward quality, domain-specific alignment effectiveness, robustness against UGC quality, and theory of mind capabilities. Our code and dataset are available at https://zhaoxuan.info/PUGC.github.io/
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）；Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系， 威特沃特斯兰大学）
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation（微软公司）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Southern California（南加州大学）
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Evaluating and Aligning CodeLLMs on Human Preference
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University（北京大学计算机科学系）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；National Research and Engineering Center of Software Engineering, Peking University（北京大学软件工程研究中心）
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）
提出自参考 AI 反馈框架，让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据，显著提升强化学习对齐效果。
ERA-IT: Aligning Semantic Models with Revealed Economic Preference for Real-Time and Explainable Patent Valuation
School of Computing（计算学院）；Macquarie University（麦考瑞大学）；School of Computer Science（计算机科学学院）；University of Technology Sydney（悉尼技术大学）；School of Management（管理学院）；Zhejiang University（浙江大学）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程保罗·G·艾伦学校）；Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根马克斯·普朗克智能系统研究所）；Jinesis Lab, University of Toronto & Vector Institute（多伦多大学Jinesis实验室及向量研究所）
