Preference alignment is pivotal for empowering large language models (LLMs) to generate helpful and harmless responses. However, the performance of preference alignment is highly sensitive to the prevalent noise in the preference data. Recent efforts for this problem either marginally alleviate the impact of noise without the ability to actually reduce its presence, or rely on costly teacher LLMs prone to reward misgeneralization. To address these challenges, we propose the RObust Preference Optimization (ROPO) framework, an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Specifically, ROPO iteratively solves a constrained optimization problem, where we dynamically assign a quality-aware weight for each sample and constrain the sum of the weights to the number of samples we intend to retain. For noise-tolerant training and effective noise identification, we derive a robust loss by suppressing the gradients of samples with high uncertainty. We demonstrate both empirically and theoretically that the derived loss is critical for distinguishing noisy samples from clean ones. Furthermore, inspired by our derived loss, we propose a robustness-guided rejection sampling technique to compensate for the potential important information in discarded queries. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods, with its superiority growing as the noise rate increases.
Weights-Rotated Preference Optimization for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
Centre for Machine Intelligence and Data Science, IIT Bombay(印度理工学院班加罗尔机器智能与数据科学中心);Koita Centre for Digital Health, IIT Bombay(印度理工学院班加罗尔数字健康中心);Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔计算机科学与工程系)
P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology(广东省计算人工智能模型与认知智能重点实验室,计算机科学与工程学院,华南理工大学);Pazhou Lab, Guangzhou, China(琶洲实验室,广州,中国);Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China(教育部健康智能感知与并行数字人工程研究中心,广州,中国)
TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
East China Normal University(东华大学);Alibaba Group(阿里巴巴集团);University of Electronic Science and Technology of China(电子科技大学);Wuhan University(武汉大学);Sun Yat-sen University(中山大学)
本文提出树引导偏好优化(TGPO)框架,通过树结构轨迹表示、过程奖励模型和动态加权机制,解决 Web Agent 强化学习中的信用分配错位、标注成本高和奖励稀疏问题,并在两个数据集上取得更高成功率和更少冗余步骤。
Department of Statistics(统计系);LSE London, UK(伦敦大学学院);Department of Mathematics(数学系);Tsinghua University(清华大学);School of Design LCC, UAL London, UK(伦敦艺术大学设计学院);Department of Engineering Science(工程科学系);University of Oxford(牛津大学)
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
Shenzhen International Center for Industrial and Applied Mathematics(深圳工业与应用数学国际中心);Shenzhen Research Institute of Big Data(深圳大数据研究 institute);The Chinese University of Hong Kong Shenzhen(香港中文大学(深圳));Birmingham City University(伯明翰城市大学);Jilin University(吉林大学);KAUST(科威特大学);Hefei University of Technology(合肥工业大学)
Adversarial Preference Learning for Robust LLM Alignment
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)技术有限公司);University of Auckland(奥克兰大学);China Telecom Corporation Limited Beijing Research Institute(中国电信股份有限公司北京研究院)
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen(计算与智能学院,哈尔滨工业大学,深圳);Huawei Cloud Computing Technologies Co., Ltd.(华为云计算技术有限公司);School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen(智能科学与工程学院,哈尔滨工业大学,深圳)
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Tongyi Lab(通义实验室)
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA(计算机科学系, Cranberry-Lemon 大学);Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa(计算神经科学系, 威特沃特斯兰大学)
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, the University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);King’s College London(伦敦国王学院);Beijing Institute for General AI(北京通用人工智能研究院);Shanghai Jiao Tong University(上海交通大学)