对扩散模型进行噪声条件感知对齐
Aligning Diffusion Models with Noise-Conditioned Perception
AI总结：
本研究提出在扩散模型的U-Net嵌入空间中使用感知目标，以提升人类偏好对齐的效率和质量，并减少计算成本。
摘要：
最近在人类偏好优化方面的进展最初是为语言模型（LMs）开发的，已显示出在文本到图像扩散模型中的前景，增强了提示对齐、视觉吸引力和用户偏好。与LMs不同，扩散模型通常在像素或VAE空间中进行优化，这与人类感知不匹配，导致在偏好对齐阶段训练较慢且效率低下。我们提出在扩散模型的U-Net嵌入空间中使用感知目标来解决这些问题。我们的方法涉及使用直接偏好优化（DPO）、对比偏好优化（CPO）和监督微调（SFT）对Stable Diffusion 1.5和XL进行微调。这种方法在各种指标上均显著优于标准潜在空间实现，包括质量和计算成本。对于SDXL，我们的方法在PartiPrompts数据集上提供了60.8%的一般偏好、62.2%的视觉吸引力和52.1%的提示遵循率，同时显著降低了计算量。我们的方法不仅提高了扩散模型的人类偏好对齐的效率和质量，而且可以与其他优化技术轻松整合。训练代码和LoRA权重将在此处提供：https://huggingface.co/alexgambashidze/SDXL_NCP-DPO_v0.1
英文摘要：
Recent advancements in human preference optimization, initially developed for Language Models (LMs), have shown promise for text-to-image Diffusion Models, enhancing prompt alignment, visual appeal, and user preference. Unlike LMs, Diffusion Models typically optimize in pixel or VAE space, which does not align well with human perception, leading to slower and less efficient training during the preference alignment stage. We propose using a perceptual objective in the U-Net embedding space of the diffusion model to address these issues. Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and supervised fine-tuning (SFT) within this embedding space. This method significantly outperforms standard latent-space implementations across various metrics, including quality and computational cost. For SDXL, our approach provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt following against original open-sourced SDXL-DPO on the PartiPrompts dataset, while significantly reducing compute. Our approach not only improves the efficiency and quality of human preference alignment for diffusion models but is also easily integrable with other optimization techniques. The training code and LoRA weights will be available here: https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
