arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.14857cs.CVcs.AIcs.LG
收藏

基于网络规模图像对的条件扩散可生成多样化图像变体

Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variations

Google Deepmind(谷歌DeepMind)

Manoj Kumar, Neil Houlsby, Emiel Hoogeboom

AI总结:

本文提出 Semantica,利用网络规模图像对进行条件扩散预训练以生成语义一致且多样的图像变体,并分析编码器设计、数据集自适应及评估指标。

摘要:

生成图像变体,即模型在保留语义上下文的同时生成输入图像的不同版本,正受到越来越多关注。当前的图像变体技术涉及调整文本到图像模型,使其在以同一图像为条件时重建输入图像。我们首先证明,训练为从冻结嵌入重建输入图像的扩散模型,能够以微小变化重建该图像。其次,受文本到图像模型如何从网络规模的文本-图像对中学习的启发,我们探索了一种新的预训练策略,利用大量图像对集合生成图像变体。我们的扩散模型 Semantica 接收来自网页的随机(编码)图像作为条件输入,并对来自同一网页的另一张带噪随机图像进行去噪。鉴于图像编码器在从输入图像提取相关上下文方面的关键作用,我们仔细考察了图像编码器的各种设计选择。训练完成后,Semantica 只需将某数据集中的图像作为输入,即可自适应地从该数据集生成新图像。最后,我们指出了用于评估图像变体的标准图像一致性指标的局限性,并提出了基于少样本生成的替代指标。

英文摘要:

Generating image variations, where a model produces variations of an input image while preserving the semantic context has gained increasing attention. Current image variation techniques involve adapting a text-to-image model to reconstruct an input image conditioned on the same image. We first demonstrate that a diffusion model trained to reconstruct an input image from frozen embeddings, can reconstruct the image with minor variations. Second, inspired by how text-to-image models learn from web-scale text-image pairs, we explore a new pretraining strategy to generate image variations using a large collection of image pairs. Our diffusion model \textit{Semantica} receives a random (encoded) image from a webpage as conditional input and denoises another noisy random image from the same webpage. We carefully examine various design choices for the image encoder, given its crucial role in extracting relevant context from the input image. Once trained, \textit{Semantica} can adaptively generate new images from a dataset by simply using images from that dataset as input. Finally, we identify limitations in standard image consistency metrics for evaluating image variations and propose alternative metrics based on few-shot generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑