DeltaSpace: 一种语义对齐的特征空间用于灵活的文本引导图像编辑
DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing
AI总结：
DeltaSpace通过语义对齐的特征空间实现文本引导图像编辑的灵活训练和推理，支持零样本推理和无需文本的训练。
摘要：
文本引导的图像编辑在考虑训练和推理灵活性时面临重大挑战。大量文献收集大量的标注图像-文本对来训练文本条件生成模型，这既昂贵又不高效。之后，一些方法利用预训练的视觉-语言模型来避免数据收集，但它们受到每文本提示优化或推理时超参数调优的限制。为了解决这些问题，我们研究并识别了一个特定的空间，称为CLIP DeltaSpace，其中两个图像的CLIP视觉特征差异在语义上与它们对应的文本描述的CLIP文本特征差异对齐。基于DeltaSpace，我们提出了一种新的框架，称为DeltaEdit，该框架在训练阶段将CLIP视觉特征差异映射到生成模型的潜在空间方向，并在推理阶段从CLIP文本特征差异预测潜在空间方向。这种设计使DeltaEdit具有两个优势：（1）无需文本的训练；（2）能够推广到各种文本提示进行零样本推理。广泛的实验验证了DeltaEdit在不同生成模型（包括GAN模型和扩散模型）中的有效性和通用性，在实现灵活的文本引导图像编辑方面取得了成果。代码可在https://github.com/Yueming6568/DeltaEdit上获得。
英文摘要：
Text-guided image editing faces significant challenges when considering training and inference flexibility. Much literature collects large amounts of annotated image-text pairs to train text-conditioned generative models from scratch, which is expensive and not efficient. After that, some approaches that leverage pre-trained vision-language models have been proposed to avoid data collection, but they are limited by either per text-prompt optimization or inference-time hyper-parameters tuning. To address these issues, we investigate and identify a specific space, referred to as CLIP DeltaSpace, where the CLIP visual feature difference of two images is semantically aligned with the CLIP textual feature difference of their corresponding text descriptions. Based on DeltaSpace, we propose a novel framework called DeltaEdit, which maps the CLIP visual feature differences to the latent space directions of a generative model during the training phase, and predicts the latent space directions from the CLIP textual feature differences during the inference phase. And this design endows DeltaEdit with two advantages: (1) text-free training; (2) generalization to various text prompts for zero-shot inference. Extensive experiments validate the effectiveness and versatility of DeltaEdit with different generative models, including both the GAN model and the diffusion model, in achieving flexible text-guided image editing. Code is available at https://github.com/Yueming6568/DeltaEdit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
