通过特征解耦与增强真实感采样方法提升扩散模型的可控性
Enhanced Controllability of Diffusion Models via Feature Disentanglement and Realism-Enhanced Sampling Methods
AI总结：
本文提出FDiff训练框架实现扩散模型特征解耦，并通过推广可组合扩散模型与时间步相关权重调度两种采样方法，在图像编辑与转换中提升真实感和可控性。
摘要：
随着扩散模型展现出良好的性能，人们为提升扩散模型的可控性付出了大量努力。然而，如何训练扩散模型使其具有解耦的潜在空间，以及如何在采样过程中自然地融入解耦条件，尚未得到充分探索。本文提出了一个用于扩散模型特征解耦的训练框架（FDiff）。我们进一步提出了两种采样方法，既能提升扩散模型的真实感，又能增强可控性。具体而言，我们以两个潜在特征为条件训练扩散模型：一个空间内容掩码和一个扁平化的风格嵌入。我们依赖扩散模型去噪过程的归纳偏置，在内容特征中编码姿态/布局信息，在风格特征中编码语义/风格信息。关于采样方法，我们首先通过打破条件独立性假设来推广可组合扩散模型（GCDM），允许条件输入之间存在一定的依赖关系，实验表明这在真实感生成中是有效的。其次，我们提出了针对内容特征和风格特征的时间步相关权重调度，以进一步提升性能。我们还观察到，与现有方法相比，我们提出的方法在图像编辑和图像转换中具有更好的可控性。
英文摘要：
As Diffusion Models have shown promising performance, a lot of efforts have been made to improve the controllability of Diffusion Models. However, how to train Diffusion Models to have the disentangled latent spaces and how to naturally incorporate the disentangled conditions during the sampling process have been underexplored. In this paper, we present a training framework for feature disentanglement of Diffusion Models (FDiff). We further propose two sampling methods that can boost the realism of our Diffusion Models and also enhance the controllability. Concisely, we train Diffusion Models conditioned on two latent features, a spatial content mask, and a flattened style embedding. We rely on the inductive bias of the denoising process of Diffusion Models to encode pose/layout information in the content feature and semantic/style information in the style feature. Regarding the sampling methods, we first generalize Composable Diffusion Models (GCDM) by breaking the conditional independence assumption to allow for some dependence between conditional inputs, which is shown to be effective in realistic generation in our experiments. Second, we propose timestep-dependent weight scheduling for content and style features to further improve the performance. We also observe better controllability of our proposed methods compared to existing methods in image manipulation and image translation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
