arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04800cs.CV
收藏

面向文本到图像扩散模型的无分割引导方法

Segmentation-Free Guidance for Text-to-Image Diffusion Models

Qualcomm AI Research(高通人工智能研究中心)

Kambiz Azarian, Debasmit Das, Qiqi Hou, Fatih Porikli

AI总结:

本文提出无分割引导方法,利用扩散模型自身作为隐式分割网络动态调整负提示,无需重训练且不增加计算成本,在文本到图像生成中显著优于无分类器方法。

摘要:

我们提出了一种名为无分割引导(segmentation-free guidance)的新方法,专为Stable Diffusion等文本到图像扩散模型设计。该方法无需对扩散模型进行重新训练,且不增加额外计算成本。它利用扩散模型本身作为隐式分割网络,因此得名无分割引导,根据生成图像中每个图像块与提示中概念的相关性,动态调整每个图像块的负提示。我们从客观和主观两个角度对无分割引导进行了评估:客观评估使用FID、CLIP、IS和PickScore指标,主观评估则通过人类评估者进行。对于主观评估,我们还提出了一种对MS COCO-30K等数据集中的提示进行子采样的方法,以在保持人类评估数量可控的同时,确保所选子集在内容上具有代表性,并在模型性能方面保持公平。结果表明,我们的无分割引导方法优于广泛使用的无分类器(classifier-free)方法。人类评估者在60%的情况下偏好无分割引导,而在19%的情况下偏好无分类器方法,另有18%的情况表现出强烈偏好。此外,PickScore胜率这一最近提出的模拟人类偏好的指标也表明,我们的方法优于无分类器方法。

英文摘要:

We introduce segmentation-free guidance, a novel method designed for text-to-image diffusion models like Stable Diffusion. Our method does not require retraining of the diffusion model. At no additional compute cost, it uses the diffusion model itself as an implied segmentation network, hence named segmentation-free guidance, to dynamically adjust the negative prompt for each patch of the generated image, based on the patch's relevance to concepts in the prompt. We evaluate segmentation-free guidance both objectively, using FID, CLIP, IS, and PickScore, and subjectively, through human evaluators. For the subjective evaluation, we also propose a methodology for subsampling the prompts in a dataset like MS COCO-30K to keep the number of human evaluations manageable while ensuring that the selected subset is both representative in terms of content and fair in terms of model performance. The results demonstrate the superiority of our segmentation-free guidance to the widely used classifier-free method. Human evaluators preferred segmentation-free guidance over classifier-free 60% to 19%, with 18% of occasions showing a strong preference. Additionally, PickScore win-rate, a recently proposed metric mimicking human preference, also indicates a preference for our method over classifier-free.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑