BootPIG：在预训练扩散模型中自举零样本个性化图像生成能力
BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models
AI总结：
本文提出 BootPIG 架构，通过独立 UNet 和合成数据训练，在约 1 小时内为预训练扩散模型赋予零样本个性化图像生成能力，效果优于现有零样本方法。
摘要：
近期的文本到图像生成模型在生成忠实遵循输入提示的图像方面取得了令人瞩目的成功。然而，使用文字描述所需概念的要求，对生成概念外观的控制仍然有限。在本工作中，我们通过提出一种使现有文本到图像扩散模型具备个性化能力的方法来解决这一缺陷。我们提出了一种新颖架构（BootPIG），允许用户提供某个对象的参考图像，以引导生成图像中概念的外观。所提出的 BootPIG 架构对预训练文本到图像扩散模型只进行最小修改，并利用一个独立的 UNet 模型引导生成结果趋向所需外观。我们引入了一种训练流程，使我们能够利用由预训练文本到图像模型、LLM 聊天智能体和图像分割模型生成的数据，在 BootPIG 架构中自举个性化能力。与需要数天预训练的现有方法不同，BootPIG 架构可在约 1 小时内完成训练。在 DreamBooth 数据集上的实验表明，BootPIG 优于现有零样本方法，同时与测试时微调方法具有可比性。通过一项用户研究，我们验证了在保持对参考对象外观的保真度以及与文本提示对齐这两方面，用户均更偏好 BootPIG 的生成结果而非现有方法。
英文摘要：
Recent text-to-image generation models have demonstrated incredible success in generating images that faithfully follow input prompts. However, the requirement of using words to describe a desired concept provides limited control over the appearance of the generated concepts. In this work, we address this shortcoming by proposing an approach to enable personalization capabilities in existing text-to-image diffusion models. We propose a novel architecture (BootPIG) that allows a user to provide reference images of an object in order to guide the appearance of a concept in the generated images. The proposed BootPIG architecture makes minimal modifications to a pretrained text-to-image diffusion model and utilizes a separate UNet model to steer the generations toward the desired appearance. We introduce a training procedure that allows us to bootstrap personalization capabilities in the BootPIG architecture using data generated from pretrained text-to-image models, LLM chat agents, and image segmentation models. In contrast to existing methods that require several days of pretraining, the BootPIG architecture can be trained in approximately 1 hour. Experiments on the DreamBooth dataset demonstrate that BootPIG outperforms existing zero-shot methods while being comparable with test-time finetuning approaches. Through a user study, we validate the preference for BootPIG generations over existing methods both in maintaining fidelity to the reference object's appearance and aligning with textual prompts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
