发现扩散模型语义潜空间中的可解释方向
Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models
AI总结：
本文探索扩散模型h-空间的语义方向，提出无监督与监督方法发现可解释方向，无需架构修改或微调，实现语义编辑。
摘要：
去噪扩散模型（DDMs）已成为生成对抗网络（GANs）的有力竞争者。然而，尽管它们在图像合成和编辑应用中广泛使用，其潜空间仍未得到充分理解。最近，一种针对DDMs的语义潜空间，被称为“$h$-空间”，被证明能够以类似于GANs的方式促进语义图像编辑。$h$-空间由DDM去噪器中扩散过程所有时间步的瓶颈激活组成。在本文中，我们探讨了h-空间的性质，并提出了几种在其中寻找有意义语义方向的新方法。我们首先研究在预训练DDMs中揭示可解释语义方向的无监督方法。具体来说，我们展示了全局潜在方向作为潜空间中的主成分出现。此外，我们提供了一种新方法，通过对去噪器相对于潜码的雅可比矩阵进行谱分析，来发现图像特定的语义方向。接下来，我们通过在无条件DDMs中以监督方式寻找方向来扩展分析。我们展示了如何依赖于真实图像的标记数据集或使用领域特定属性分类器对生成的样本进行标注来找到这样的方向。我们还进一步展示了如何通过简单的线性投影在语义上解纠缠所找到的方向。我们的方法无需任何架构修改、基于文本的引导、基于CLIP的优化或模型微调即可应用。
英文摘要：
Denoising Diffusion Models (DDMs) have emerged as a strong competitor to Generative Adversarial Networks (GANs). However, despite their widespread use in image synthesis and editing applications, their latent space is still not as well understood. Recently, a semantic latent space for DDMs, coined `$h$-space', was shown to facilitate semantic image editing in a way reminiscent of GANs. The $h$-space is comprised of the bottleneck activations in the DDM's denoiser across all timesteps of the diffusion process. In this paper, we explore the properties of h-space and propose several novel methods for finding meaningful semantic directions within it. We start by studying unsupervised methods for revealing interpretable semantic directions in pretrained DDMs. Specifically, we show that global latent directions emerge as the principal components in the latent space. Additionally, we provide a novel method for discovering image-specific semantic directions by spectral analysis of the Jacobian of the denoiser w.r.t. the latent code. Next, we extend the analysis by finding directions in a supervised fashion in unconditional DDMs. We demonstrate how such directions can be found by relying on either a labeled data set of real images or by annotating generated samples with a domain-specific attribute classifier. We further show how to semantically disentangle the found direction by simple linear projection. Our approaches are applicable without requiring any architectural modifications, text-based guidance, CLIP-based optimization, or model fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
