The success of AI assistants based on language models (LLMs) hinges crucially on Reinforcement Learning from Human Feedback (RLHF), which enables the generation of responses more aligned with human preferences. As universal AI assistants, there's a growing expectation for them to perform consistently across various domains. However, previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. This focus on quick reward gains undermines both the stability in training and the model's ability to generalize to new, unseen data. In this work, we propose a novel approach that can learn a consistent policy via RL across various data groups or domains. Given the challenges associated with acquiring group annotations, our method automatically classifies data into different groups, deliberately maximizing performance variance. Then, we optimize the policy to perform well on challenging groups. Lastly, leveraging the established groups, our approach adaptively adjusts the exploration space, allocating more learning capacity to more challenging data and preventing the model from over-optimizing on simpler data. Experimental results indicate that our approach significantly enhances training stability and model generalization.
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University(国立台湾大学);Texas A&M University(德克萨斯A&M大学);Appier AI Research(Appier人工智能研究院);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所)
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India;Department of Computing Science, \ of Alberta Canada;Qatar Computing Research Institute, \ Bin Khalifa University Qatar;Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India;TCS Research, \ of CSE, IIT Madras;Department of Computing Science, \ of Alberta;Qatar Computing Research Institute, \ Bin Khalifa University;Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);King’s College London(伦敦国王学院);Beijing Institute for General AI(北京通用人工智能研究院);Shanghai Jiao Tong University(上海交通大学)
Harvard University(哈佛大学);University of California Berkeley(加州大学伯克利分校);Massachusetts Institute of Technology(麻省理工学院);Ikigai Labs(Ikigai实验室);University of California San Diego(加州大学圣地亚哥分校)
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
Tsinghua University(清华大学);International Digital Economy Academy(国际数字经济学院);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Nanyang Technological University(南洋理工大学)
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学计算机科学系);Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学电气与计算机工程系)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
Graduate School of AI, POSTECH, Pohang, Republic of Korea(POSTECH人工智能研究生院);Department of CSE, POSTECH, Pohang, Republic of Korea(POSTECH计算机科学与工程系);National AI Research Lab, Seoul, Republic of Korea(首尔国家人工智能研究实验室)
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool(利物浦大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Edinburgh(爱丁堡大学);University of Bristol(布里斯托大学);University of Exeter(埃克塞特大学)
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Harbin Institute of Technology(哈尔滨工业大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Hong Kong University of Science and Technology(香港科技大学)
Data Selection for LLM Alignment Using Fine-Grained Preferences
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Algorithm Tech, Taobao & Tmall Group of Alibaba(阿里巴巴集团算法技术部);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)