FRAP:基于自适应提示加权的忠实且逼真的文本到图像生成
FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting
AI总结:
本文提出FRAP方法,通过在线算法自适应调整每个标记的提示权重,最小化统一目标函数以增强对象存在和修饰符绑定,从而在更低延迟下显著提升文本到图像生成的提示对齐度和图像真实性。
摘要:
文本到图像(T2I)扩散模型在给定文本提示下生成高质量图像方面展现了令人印象深刻的能力。然而,确保提示-图像对齐仍然是一个相当大的挑战,即生成忠实符合提示语义的图像。最近的工作试图通过优化潜在代码来提高忠实度,这可能导致潜在代码偏离分布,从而产生不真实的图像。在本文中,我们提出了FRAP,一种简单而有效的方法,基于自适应调整每个标记的提示权重来改善提示-图像对齐和生成图像的真实性。我们设计了一个在线算法来自适应更新每个标记的权重系数,这是通过最小化一个统一的目标函数实现的,该函数鼓励对象存在以及对象-修饰符对的绑定。通过广泛的评估,我们表明FRAP在复杂数据集的提示下生成的图像具有显著更高的提示-图像对齐度,同时与最近的潜在代码优化方法相比具有更低的平均延迟,例如,在COCO-Subject数据集上比D&B快4秒。此外,通过视觉比较和对CLIP-IQA-Real指标的评估,我们表明FRAP不仅改善了提示-图像对齐,还生成了外观更真实的图像。我们还探索将FRAP与提示重写LLM结合以恢复其退化的提示-图像对齐,观察到提示-图像对齐和图像质量均有改善。我们在以下链接发布代码:https://github.com/LiyaoJiang1998/FRAP/。
英文摘要:
Text-to-image (T2I) diffusion models have demonstrated impressive capabilities in generating high-quality images given a text prompt. However, ensuring the prompt-image alignment remains a considerable challenge, i.e., generating images that faithfully align with the prompt's semantics. Recent works attempt to improve the faithfulness by optimizing the latent code, which potentially could cause the latent code to go out-of-distribution and thus produce unrealistic images. In this paper, we propose FRAP, a simple, yet effective approach based on adaptively adjusting the per-token prompt weights to improve prompt-image alignment and authenticity of the generated images. We design an online algorithm to adaptively update each token's weight coefficient, which is achieved by minimizing a unified objective function that encourages object presence and the binding of object-modifier pairs. Through extensive evaluations, we show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets, while having a lower average latency compared to recent latent code optimization methods, e.g., 4 seconds faster than D&B on the COCO-Subject dataset. Furthermore, through visual comparisons and evaluation of the CLIP-IQA-Real metric, we show that FRAP not only improves prompt-image alignment but also generates more authentic images with realistic appearances. We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment, where we observe improvements in both prompt-image alignment and image quality. We release the code at the following link: https://github.com/LiyaoJiang1998/FRAP/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。