Reinforcement learning (RL) algorithms have been used recently to align diffusion models with downstream objectives such as aesthetic quality and text-image consistency by fine-tuning them to maximize a single reward function under a fixed KL regularization. However, this approach is inherently restrictive in practice, where alignment must balance multiple, often conflicting objectives. Moreover, user preferences vary across prompts, individuals, and deployment contexts, with varying tolerances for deviation from a pre-trained base model. We address the problem of inference-time multi-preference alignment: given a set of basis reward functions and a reference KL regularization strength, can we design a fine-tuning procedure so that, at inference time, it can generate images aligned with any user-specified linear combination of rewards and regularization, without requiring additional fine-tuning? We propose Diffusion Blend, a novel approach to solve inference-time multi-preference alignment by blending backward diffusion processes associated with fine-tuned models, and we instantiate this approach with two algorithms: DB-MPA for multi-reward alignment and DB-KLA for KL regularization control. Extensive experiments show that Diffusion Blend algorithms consistently outperform relevant baselines and closely match or exceed the performance of individually fine-tuned models, enabling efficient, user-driven alignment at inference-time. The code is available at https://github.com/bluewoods127/DB-2025.
Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(人工智能向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Kling Team, Kuaishou Technology(快手科技 Kling 团队);NSERC CRC Chair(加拿大NSERC CRC主席)
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool(利物浦大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Edinburgh(爱丁堡大学);University of Bristol(布里斯托大学);University of Exeter(埃克塞特大学)
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India;Department of Computing Science, \ of Alberta Canada;Qatar Computing Research Institute, \ Bin Khalifa University Qatar;Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India;TCS Research, \ of CSE, IIT Madras;Department of Computing Science, \ of Alberta;Qatar Computing Research Institute, \ Bin Khalifa University;Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy
Nankai University(南开大学);Tsinghua University(清华大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Wuhan University(武汉大学)
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用关键实验室);Ministry of Education(教育部)
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学计算机科学系);Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学电气与计算机工程系)