BLIP-Diffusion：用于可控文本到图像生成与编辑的预训练主体表示
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
AI总结：
BLIP-Diffusion通过预训练多模态编码器提供主体表示，实现零样本主体驱动生成和高效微调，速度提升20倍，并可与ControlNet等结合用于生成和编辑。
摘要：
主体驱动的文本到图像生成模型基于文本提示创建输入主体的新颖渲染。现有模型存在微调时间长和难以保持主体保真度的问题。为克服这些限制，我们引入了BLIP-Diffusion，一种新的主体驱动图像生成模型，支持多模态控制，即同时输入主体图像和文本提示。与其他主体驱动生成模型不同，BLIP-Diffusion引入了一个新的多模态编码器，该编码器经过预训练以提供主体表示。我们首先按照BLIP-2预训练多模态编码器，以产生与文本对齐的视觉表示。然后，我们设计了一个主体表示学习任务，使扩散模型能够利用这种视觉表示并生成新的主体渲染。与DreamBooth等先前方法相比，我们的模型实现了零样本主体驱动生成，以及针对定制主体的高效微调，速度提升高达20倍。我们还证明了BLIP-Diffusion可以灵活地与现有技术（如ControlNet和prompt-to-prompt）结合，以实现新颖的主体驱动生成和编辑应用。代码和模型将在https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion发布。项目页面在https://dxli94.github.io/BLIP-Diffusion-website/。
英文摘要：
Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Code and models will be released at https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion. Project page at https://dxli94.github.io/BLIP-Diffusion-website/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
