Despite the fact that offline methods for Large Language Models (LLMs) alignment do not require a direct reward model, they remain susceptible to overoptimization. This issue arises when the trained model deviates excessively from the reference policy, leading to a decrease in sample quality. We propose a new paradigm of offline alignment methods, called Trust Region (including variants TR-DPO, TR-IPO, TR-KTO), which dynamically updates the reference policy throughout the training process. Our results show that TR alignment methods effectively mitigate overoptimization, enabling models to maintain strong performance even when substantially deviating from the initial reference policy. We demonstrate the efficacy of these approaches not only through toy examples that exhibit reduced overoptimization, but also through direct, side-by-side comparisons in specific tasks such as helpful and harmless dialogue, as well as summarization, where they surpass conventional methods. Additionally, we report significant improvements in general-purpose assistant setups with the Llama3 model on the AlpacaEval 2 and Arena-Hard benchmarks, highlighting the advantages of Trust Region methods over classical approaches.
Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
School of Life Sciences, IDG/McGovern Institute for Brain Research(生命科学学院,IDG/麦戈文脑科学研究所);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center(计算机科学与技术系,人工智能研究所,清华大学-博世联合机器学习中心,THBI实验室,BNRist中心);Tsinghua University, Beijing, China(清华大学,北京,中国)
The Greatest Good Benchmark: Measuring LLMs' Alignment with Utilitarian Moral Dilemmas
Universidad de Buenos Aires(布宜诺斯艾利斯大学);Universidad Torcuato Di Tella(托库阿托·迪·特拉大学);Lumina Labs(卢米纳实验室);Facultad de Ciencias Exactas y Naturales(精确与自然科学学院);Escuela de Negocios(商学院)
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
ShanghaiTech University(上海科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Meituan(美团);Alibaba Group(阿里巴巴集团);The Hong Kong University of Science and Technology(香港科技大学)
Graph Alignment Topology as an Inductive Bias for Grounding Detection
Department of Pediatrics, University of Illinois College of Medicine Peoria(伊利诺伊大学皮奥里亚医学院儿科部);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机与数据科学学院);Carle Illinois College of Medicine, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校卡莱医学院)
How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
Inria(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学校);PSL Research University(巴黎综合理工研究所);Northwestern University(西北大学);University of California, Berkeley(加州大学伯克利分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Mitigating Mismatch within Reference-based Preference Optimization
Sydney AI Centre, The University of Sydney(悉尼人工智能中心,悉尼大学);CFAR, A*STAR(CFAR,A*STAR);CSIRO, Data61(CSIRO,Data61);Southeast University(东南大学);Nanyang Technological University(南洋理工大学)
Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matching
Joint Laboratory of Data Science and Business Intelligence(数据科学与商业智能联合实验室);State Key Laboratory of Mathematical Sciences(数学科学国家重点实验室);Institut für Geometrie und Praktische Mathematik(几何与应用数学研究所);School of Artificial Intelligence(人工智能学院);School of Mathematics and Statistics(数学与统计学院)
Department of Statistics(统计系);LSE London, UK(伦敦大学学院);Department of Mathematics(数学系);Tsinghua University(清华大学);School of Design LCC, UAL London, UK(伦敦艺术大学设计学院);Department of Engineering Science(工程科学系);University of Oxford(牛津大学)