扩散模型已在文本到图像生成中取得显著成功,这使得对这些模型进行偏好对齐变得日益重要。偏好标签通常只在去噪轨迹的末端可用,这给优化中间去噪步骤带来了挑战。本文提出进行去噪分布估计(Denoised Distribution Estimation,DDE),该方法显式地将中间步骤与末端去噪分布联系起来。因此,偏好标签可用于整个轨迹的优化。为此,我们为DDE设计了两种估计策略。第一种是逐步估计,它利用条件去噪分布来估计模型去噪分布。第二种是单次估计,它通过DDIM建模将模型输出转换为末端去噪分布。在分析和实验上,我们揭示了配备两种估计策略的DDE会自然推导出一种新颖的信用分配方案,该方案优先优化去噪轨迹的中间部分。大量实验表明,我们的方法在定量和定性上均取得了优越性能。
英文摘要:
Diffusion models have shown remarkable success in text-to-image generation, making preference alignment for these models increasingly important. The preference labels are typically available only at the terminal of denoising trajectories, which poses challenges in optimizing the intermediate denoising steps. In this paper, we propose to conduct Denoised Distribution Estimation (DDE) that explicitly connects intermediate steps to the terminal denoised distribution. Therefore, preference labels can be used for the entire trajectory optimization. To this end, we design two estimation strategies for our DDE. The first is stepwise estimation, which utilizes the conditional denoised distribution to estimate the model denoised distribution. The second is single-shot estimation, which converts the model output into the terminal denoised distribution via DDIM modeling. Analytically and empirically, we reveal that DDE equipped with two estimation strategies naturally derives a novel credit assignment scheme that prioritizes optimizing the middle part of the denoising trajectory. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool(利物浦大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Edinburgh(爱丁堡大学);University of Bristol(布里斯托大学);University of Exeter(埃克塞特大学)
Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(人工智能向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Kling Team, Kuaishou Technology(快手科技 Kling 团队);NSERC CRC Chair(加拿大NSERC CRC主席)
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Harbin Institute of Technology(哈尔滨工业大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Hong Kong University of Science and Technology(香港科技大学)
Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(智能机器人与先进制造学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University(上海智能信息处理重点实验室,计算机科学与人工智能学院,复旦大学);Peking University(北京大学);vivo Mobile Communication Co., Ltd(vivo移动通信有限公司)
Low-dose CT Denoising with Language-engaged Dual-space Alignment
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学脑智能科学与技术研究院);Center for Biotechnology and Interdisciplinary Studies, Rensselaer Polytechnic Institute(伦斯勒理工学院生物技术与跨学科研究中心)