Instruct-CLIP：利用对比学习通过自动化数据精炼改进指令引导的图像编辑
Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning
AI总结：
针对指令引导图像编辑中训练数据与指令不对齐的问题，提出自监督方法Instruct-CLIP，通过对比学习精炼数据集并用于训练潜在扩散模型，显著提升编辑结果与指令的一致性。
摘要：
尽管自然语言指令为引导自动化图像编辑提供了一种直观的方式，但深度学习模型往往难以获得高质量的结果，这主要归因于创建大规模高质量训练数据集的困难。为此，以往的方法通常依赖文本到图像（T2I）生成模型来生成原始图像和编辑后图像对，以模拟指令引导图像编辑模型的输入/输出。然而，由于T2I模型的局限性，这些图像对往往无法与指定的编辑指令对齐，从而对基于此类数据集训练的模型产生负面影响。为解决这一问题，我们提出了Instruct-CLIP（I-CLIP），一种自监督方法，它学习原始图像与编辑后图像之间的语义变化，以精炼并更好地对齐现有数据集中的指令。此外，我们调整Instruct-CLIP以处理带噪声的潜在图像和扩散时间步长，使其可用于训练潜在扩散模型（LDMs），并在扩散流程的任何步骤中高效地强制编辑指令与潜在空间中的图像变化之间的对齐。我们使用Instruct-CLIP来校正InstructPix2Pix数据集，获得超过12万个精炼样本，然后利用这些样本，并借助我们基于I-CLIP的新型损失函数来微调其模型。由此产生的模型能够生成与给定指令更一致的编辑结果。我们的代码和数据集可在 https://github.com/SherryXTChen/Instruct-CLIP.git 获取。
英文摘要：
Although natural language instructions offer an intuitive way to guide automated image editing, deep-learning models often struggle to achieve high-quality results, largely due to the difficulty of creating large, high-quality training datasets. To do this, previous approaches have typically relied on text-to-image (T2I) generative models to produce pairs of original and edited images that simulate the input/output of an instruction-guided image-editing model. However, these image pairs often fail to align with the specified edit instructions due to the limitations of T2I models, which negatively impacts models trained on such datasets. To address this, we present Instruct-CLIP (I-CLIP), a selfsupervised method that learns the semantic changes between original and edited images to refine and better align the instructions in existing datasets. Furthermore, we adapt Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it can be used to train latent diffusion models (LDMs) and efficiently enforce alignment between the edit instruction and the image changes in latent space at any step of the diffusion pipeline. We use Instruct-CLIP to correct the InstructPix2Pix dataset and get over 120K refined samples we then use to fine-tune their model, guided by our novel I-CLIP-based loss function. The resulting model can produce edits that are more aligned with the given instructions. Our code and dataset are available at https://github.com/SherryXTChen/Instruct-CLIP.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
