DeltaEdit：探索文本驱动图像编辑的无文本训练方法
DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation
AI总结：
本文提出 DeltaEdit 框架，利用 CLIP 视觉与文本特征差异对齐的 delta 空间，以无文本方式训练网络预测 StyleGAN 编辑方向，实现文本驱动图像编辑的零样本泛化。
摘要：
文本驱动的图像编辑在训练或推理灵活性方面仍然具有挑战性。条件生成模型严重依赖昂贵的标注训练数据。与此同时，近期利用预训练视觉-语言模型的框架，要么受限于逐文本提示优化，要么受限于推理时的超参数调整。本文提出了一种名为 DeltaEdit 的新框架来解决这些问题。我们的核心思想是研究并识别一个空间，即 delta 图像与文本空间，该空间在两张图像的 CLIP 视觉特征差异与源文本和目标文本的 CLIP 文本嵌入差异之间具有良好对齐的分布。基于 CLIP delta 空间，DeltaEdit 网络被设计为在训练阶段将 CLIP 视觉特征差异映射到 StyleGAN 的编辑方向。然后，在推理阶段，DeltaEdit 根据 CLIP 文本特征的差异预测 StyleGAN 的编辑方向。通过这种方式，DeltaEdit 以无文本的方式进行训练。一旦训练完成，它能够很好地泛化到各种文本提示，实现零样本推理，无需额外技巧。
英文摘要：
Text-driven image manipulation remains challenging in training or inference flexibility. Conditional generative models depend heavily on expensive annotated training data. Meanwhile, recent frameworks, which leverage pre-trained vision-language models, are limited by either per text-prompt optimization or inference-time hyper-parameters tuning. In this work, we propose a novel framework named \textit{DeltaEdit} to address these problems. Our key idea is to investigate and identify a space, namely delta image and text space that has well-aligned distribution between CLIP visual feature differences of two images and CLIP textual embedding differences of source and target texts. Based on the CLIP delta space, the DeltaEdit network is designed to map the CLIP visual features differences to the editing directions of StyleGAN at training phase. Then, in inference phase, DeltaEdit predicts the StyleGAN's editing directions from the differences of the CLIP textual features. In this way, DeltaEdit is trained in a text-free manner. Once trained, it can well generalize to various text prompts for zero-shot inference without bells and whistles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
