arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.18442cs.LGcs.CV
收藏

潜空间扩散模型中的无梯度解码器逆映射

Gradient-free Decoder Inversion in Latent Diffusion Models

Seoul National University(首尔国立大学);University of California, Los Angeles(加州大学洛杉矶分校)

Seongmin Hong, Suh Yoon Jeon, Kyeonghyun Lee, Ernest K. Ryu, Se Young Chun

AI总结:

针对潜空间扩散模型中基于梯度的解码器逆映射内存消耗大、计算慢的问题,提出一种高效的无梯度解码器逆映射方法,具有理论收敛保证,并在显著降低计算开销的同时实现了与现有方法可比的误差水平。

摘要:

在潜空间扩散模型(LDMs)中,去噪扩散过程高效地在维度低于像素空间的潜空间上进行。通常使用解码器将潜空间的表示转换为像素空间的表示。尽管假设解码器具有作为精确逆映射的编码器,但实践中极少存在精确的编码器-解码器对,即使应用通常需要精确的解码器逆映射。先前LDMs中的解码器逆映射工作采用了受生成对抗网络逆映射启发的梯度下降方法。然而,基于梯度的方法对于更大的潜空间需要更大的GPU内存和更长的计算时间。例如,最近的视频LDMs可以生成超过16帧,但24 GB内存的GPU只能对4帧执行基于梯度的解码器逆映射。在此,我们提出了一种用于LDMs的高效无梯度解码器逆映射方法,可应用于多种潜空间模型。在关于近期LDMs所满足的余强制性的温和假设下,我们不仅研究了前向步长方法的理论收敛性,还研究了惯性Krasnoselskii-Mann(KM)迭代的理论收敛性。我们提出的结合Adam优化器和学习率调度的无梯度方法,显著减少了相比先前基于梯度方法的计算时间和内存使用,并在实现可比误差水平的同时,支持了诸如噪声空间水印等应用的高效计算。

英文摘要:

In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑