arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.14871cs.CVcs.AIcs.CLcs.LG
收藏

通过显式去噪分布估计实现扩散模型的偏好对齐

Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation

Alibaba Group(阿里巴巴集团)

Dingyuan Shi, Yong Wang, Hangyu Li, Xiangxiang Chu

AI总结:

针对扩散模型偏好标签仅位于去噪轨迹末端、难以优化中间步骤的问题,本文提出DDE及逐步、单次两种估计策略,实现全轨迹偏好对齐并优先优化中段,实验表现优越。

摘要:

扩散模型已在文本到图像生成中取得显著成功,这使得对这些模型进行偏好对齐变得日益重要。偏好标签通常只在去噪轨迹的末端可用,这给优化中间去噪步骤带来了挑战。本文提出进行去噪分布估计(Denoised Distribution Estimation,DDE),该方法显式地将中间步骤与末端去噪分布联系起来。因此,偏好标签可用于整个轨迹的优化。为此,我们为DDE设计了两种估计策略。第一种是逐步估计,它利用条件去噪分布来估计模型去噪分布。第二种是单次估计,它通过DDIM建模将模型输出转换为末端去噪分布。在分析和实验上,我们揭示了配备两种估计策略的DDE会自然推导出一种新颖的信用分配方案,该方案优先优化去噪轨迹的中间部分。大量实验表明,我们的方法在定量和定性上均取得了优越性能。

英文摘要:

Diffusion models have shown remarkable success in text-to-image generation, making preference alignment for these models increasingly important. The preference labels are typically available only at the terminal of denoising trajectories, which poses challenges in optimizing the intermediate denoising steps. In this paper, we propose to conduct Denoised Distribution Estimation (DDE) that explicitly connects intermediate steps to the terminal denoised distribution. Therefore, preference labels can be used for the entire trajectory optimization. To this end, we design two estimation strategies for our DDE. The first is stepwise estimation, which utilizes the conditional denoised distribution to estimate the model denoised distribution. The second is single-shot estimation, which converts the model output into the terminal denoised distribution via DDIM modeling. Analytically and empirically, we reveal that DDE equipped with two estimation strategies naturally derives a novel credit assignment scheme that prioritizes optimizing the middle part of the denoising trajectory. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑