通过优化人类效用对齐扩散模型
Aligning Diffusion Models by Optimizing Human Utility
AI总结：
Diffusion-KTO通过最大化预期人类效用对齐文本到图像扩散模型，仅需简单的每张图像二元反馈信号，无需成对偏好数据或复杂奖励模型，在人类判断和自动评估指标上优于监督微调和Diffusion-DPO。
摘要：
我们提出了Diffusion-KTO，一种通过对齐目标表述为最大化预期人类效用来对齐文本到图像扩散模型的新方法。由于该目标独立应用于每次生成，Diffusion-KTO无需收集昂贵的成对偏好数据，也无需训练复杂的奖励模型。相反，我们的目标仅需简单的每张图像二元反馈信号，例如喜欢或不喜欢，这些信号大量存在。使用Diffusion-KTO微调后，文本到图像扩散模型在人类判断和自动评估指标（如PickScore和ImageReward）方面均表现出优于现有技术（包括监督微调和Diffusion-DPO）的性能。总体而言，Diffusion-KTO释放了利用现成每张图像二元信号的潜力，并拓宽了将文本到图像扩散模型与人类偏好对齐的适用性。
英文摘要：
We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
