arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.18547cs.AIcs.CV
收藏

扩散混合:用于扩散模型的推理时间多偏好对齐

Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models

Texas A&M University(德克萨斯大学阿姆斯特朗分校);Qualcomm AI Research(高通人工智能研究)

Min Cheng, Fatemeh Doudi, Dileep Kalathil, Mohammad Ghavamzadeh, Panganamala R. Kumar

AI总结:

扩散混合通过混合反向扩散过程实现推理时间多偏好对齐,提升用户驱动的图像生成性能。

摘要:

强化学习(RL)算法最近被用来通过微调模型以最大化单个奖励函数并在固定KL正则化下对扩散模型进行对齐,以实现下游目标,如美学质量和文本-图像一致性。然而,这种做法在实践中本质上是受限的,因为对齐必须在多个往往相互冲突的目标之间进行平衡。此外,用户偏好在提示、个人和部署上下文中各不相同,对偏离预训练基础模型的容忍度也各不相同。我们解决了推理时间多偏好对齐的问题:给定一组基础奖励函数和一个参考KL正则化强度,能否设计一种微调过程,使得在推理时间生成的图像能够与任何用户指定的奖励和正则化的线性组合对齐,而无需额外的微调?我们提出了扩散混合,一种通过混合微调模型相关的反向扩散过程来解决推理时间多偏好对齐的新方法,并通过两种算法实现了该方法:DB-MPA用于多奖励对齐和DB-KLA用于KL正则化控制。广泛的实验表明,扩散混合算法在性能上始终优于相关基线,并且与单独微调的模型表现相当或更优,从而实现了高效的、由用户驱动的推理时间对齐。代码可在https://github.com/bluewoods127/DB-2025获取。

英文摘要:

Reinforcement learning (RL) algorithms have been used recently to align diffusion models with downstream objectives such as aesthetic quality and text-image consistency by fine-tuning them to maximize a single reward function under a fixed KL regularization. However, this approach is inherently restrictive in practice, where alignment must balance multiple, often conflicting objectives. Moreover, user preferences vary across prompts, individuals, and deployment contexts, with varying tolerances for deviation from a pre-trained base model. We address the problem of inference-time multi-preference alignment: given a set of basis reward functions and a reference KL regularization strength, can we design a fine-tuning procedure so that, at inference time, it can generate images aligned with any user-specified linear combination of rewards and regularization, without requiring additional fine-tuning? We propose Diffusion Blend, a novel approach to solve inference-time multi-preference alignment by blending backward diffusion processes associated with fine-tuned models, and we instantiate this approach with two algorithms: DB-MPA for multi-reward alignment and DB-KLA for KL regularization control. Extensive experiments show that Diffusion Blend algorithms consistently outperform relevant baselines and closely match or exceed the performance of individually fine-tuned models, enabling efficient, user-driven alignment at inference-time. The code is available at https://github.com/bluewoods127/DB-2025.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑