In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pairs of preferences derived from the same prompts, and it functions without needing an additional reward model. However, DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical but also similar questions. To overcome this shortfall, we propose Relative Preference Optimization (RPO). RPO is designed to discern between more and less preferred responses derived from both identical and related prompts. It introduces a contrastive weighting mechanism, enabling the tuning of LLMs using a broader range of preference data, including both paired and unpaired sets. This approach expands the learning capabilities of the model, allowing it to leverage insights from a more varied set of prompts. Through empirical tests, including dialogue and summarization tasks, and evaluations using the AlpacaEval2.0 leaderboard, RPO has demonstrated a superior ability to align LLMs with user preferences and to improve their adaptability during the training process. Our code can be viewed at https://github.com/yinyueqin/relative-preference-optimization
Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(人工智能与数字经济广东实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Baidu Inc.(百度公司)
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
East China Normal University(东华大学);Alibaba Group(阿里巴巴集团);University of Electronic Science and Technology of China(电子科技大学);Wuhan University(武汉大学);Sun Yat-sen University(中山大学)
本文提出树引导偏好优化(TGPO)框架,通过树结构轨迹表示、过程奖励模型和动态加权机制,解决 Web Agent 强化学习中的信用分配错位、标注成本高和奖励稀疏问题,并在两个数据集上取得更高成功率和更少冗余步骤。
SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generation
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Chinese Academy of Sciences(中国科学院);Shanghai AI Lab(上海人工智能实验室);Tencent Hunyuan(腾讯文英)
Reward Models Inherit Value Biases from Pretraining
Department of Experimental Psychology, University of Oxford(心理学系,牛津大学);Department of Computer Science, University of Oxford(计算机科学系,牛津大学);AI/ML Research Group, Universitat Pompeu Fabra(人工智能/机器学习研究组,庞培法华大学);Oxford Internet Institute, University of Oxford(牛津互联网研究所,牛津大学)
College of Artificial Intelligence, MIIT Key Laboratory of Pattern Analysis(人工智能学院,模式分析与机器智能关键实验室);The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics(脑机智能技术关键实验室,教育部,南京航空航天大学)