通过可解释的潜在空间操控增强条件图像生成
Enhancing Conditional Image Generation with Explainable Latent Space Manipulation
AI总结：
本文提出结合扩散模型、潜在空间操控与 Grad-SAM 梯度选择性注意力的方法，通过在去噪特定时间步生成掩码保留主体并融合参考图像特征，在 places365 上取得最低 FID 和高 CLIP 分数，提升了文本到图像合成的保真度与文本一致性。
摘要：
在图像合成领域，在遵循条件提示的同时实现对参考图像的保真度仍然是一项重大挑战。本文提出了一种新方法，将扩散模型与潜在空间操控以及基于梯度的选择性注意力机制相结合来解决这一问题。利用 Grad-SAM（基于梯度的选择性注意力操控），我们分析了交叉注意力层的交叉注意力图以及去噪潜在向量的梯度，从而推导出去噪潜在向量中与感兴趣主体相关的元素的重要性分数。利用这些信息，我们在去噪过程中的特定时间步创建掩码，以保留主体的同时无缝整合参考图像特征。该方法确保基于条件提示忠实形成主体，同时细化背景以实现更连贯的构图。我们在 places365 数据集上的实验展示了有前景的结果，与基线模型相比，我们提出的模型取得了最低的均值和 median Frechet Inception Distance（FID）分数，表明其具有优越的保真度保持能力。此外，我们的模型在使生成图像与所提供的文本描述对齐方面表现出具有竞争力的性能，高 CLIP 分数证明了这一点。这些结果凸显了我们的方法在保真度保持和文本上下文保持方面的有效性，为文本到图像合成任务提供了一项重要进展。
英文摘要：
In the realm of image synthesis, achieving fidelity to a reference image while adhering to conditional prompts remains a significant challenge. This paper proposes a novel approach that integrates a diffusion model with latent space manipulation and gradient-based selective attention mechanisms to address this issue. Leveraging Grad-SAM (Gradient-based Selective Attention Manipulation), we analyze the cross attention maps of the cross attention layers and gradients for the denoised latent vector, deriving importance scores of elements of denoised latent vector related to the subject of interest. Using this information, we create masks at specific timesteps during denoising to preserve subjects while seamlessly integrating the reference image features. This approach ensures the faithful formation of subjects based on conditional prompts, while concurrently refining the background for a more coherent composition. Our experiments on places365 dataset demonstrate promising results, with our proposed model achieving the lowest mean and median Frechet Inception Distance (FID) scores compared to baseline models, indicating superior fidelity preservation. Furthermore, our model exhibits competitive performance in aligning the generated images with provided textual descriptions, as evidenced by high CLIP scores. These results highlight the effectiveness of our approach in both fidelity preservation and textual context preservation, offering a significant advancement in text-to-image synthesis tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
