Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.
Inference-Time Alignment for Diffusion Models via Variationally Stable Doob's Matching
Joint Laboratory of Data Science and Business Intelligence（数据科学与商业智能联合实验室）；State Key Laboratory of Mathematical Sciences（数学科学国家重点实验室）；Institut für Geometrie und Praktische Mathematik（几何与应用数学研究所）；School of Artificial Intelligence（人工智能学院）；School of Mathematics and Statistics（数学与统计学院）
LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment
Dept. of Computer Engineering（计算机工程系）；SIES Graduate School of Technology（SIES技术研究生学院）；Bharatiya Vidya Bhavan's Sardar Patel Institute of Technology（巴哈里亚·维达·巴万学院萨达尔·帕特尔技术学院）
A Statistical Framework for Alignment with Biased AI Feedback
Center for Data Science, Zhejiang University（浙江大学数据科学中心）；Department of Electrical and Computer Engineering, Rutgers University（罗格斯大学电气与计算机工程系）；Department of Statistics, Rutgers University（罗格斯大学统计学系）；Faculty of Business and Economics, The University of Hong Kong（香港大学商学院）
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Pittsburgh（匹兹堡大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
