将文本到图像生成的偏好对齐范式推广到 $f$-散度最小化
Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization
AI总结：
该研究将文本到图像模型的对齐范式从反向KL散度推广到$f$-散度，通过梯度场分析不同散度约束的影响，发现Jensen-Shannon散度在对齐性能与生成多样性间取得最佳权衡。
摘要：
直接偏好优化（DPO）最近将其成功应用从对齐大型语言模型（LLMs）扩展到根据人类偏好对齐文本到图像模型，这在社区内引起了相当大的兴趣。然而，我们观察到这些方法在对齐过程中仅依赖最小化微调模型与参考模型之间的反向Kullback-Leibler散度，而忽略了其他散度约束的纳入。在本研究中，我们专注于将文本到图像模型对齐范式中的反向Kullback-Leibler散度推广到 $f$-散度，旨在获得更好的对齐性能以及良好的生成多样性。我们提供了在 $f$-散度条件下对齐范式的广义公式，并从梯度场的角度深入分析了不同散度约束对对齐过程的影响。我们在不同散度约束下对图像-文本对齐性能、人类价值对齐性能和生成多样性性能进行了全面评估，结果表明基于Jensen-Shannon散度的对齐实现了它们之间的最佳权衡。用于对齐文本到图像模型的散度选择显著影响对齐性能（尤其是人类价值对齐）与生成多样性之间的权衡，这突显了在实际应用中选择合适散度的必要性。
英文摘要：
Direct Preference Optimization (DPO) has recently expanded its successful application from aligning large language models (LLMs) to aligning text-to-image models with human preferences, which has generated considerable interest within the community. However, we have observed that these approaches rely solely on minimizing the reverse Kullback-Leibler divergence during alignment process between the fine-tuned model and the reference model, neglecting the incorporation of other divergence constraints. In this study, we focus on extending reverse Kullback-Leibler divergence in the alignment paradigm of text-to-image models to $f$-divergence, which aims to garner better alignment performance as well as good generation diversity. We provide the generalized formula of the alignment paradigm under the $f$-divergence condition and thoroughly analyze the impact of different divergence constraints on alignment process from the perspective of gradient fields. We conduct comprehensive evaluation on image-text alignment performance, human value alignment performance and generation diversity performance under different divergence constraints, and the results indicate that alignment based on Jensen-Shannon divergence achieves the best trade-off among them. The option of divergence employed for aligning text-to-image models significantly impacts the trade-off between alignment performance (especially human value alignment) and generation diversity, which highlights the necessity of selecting an appropriate divergence for practical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
