噪声结晶与液体噪声：使用图像扩散模型的零样本视频生成
Noise Crystallization and Liquid Noise: Zero-shot Video Generation using Image Diffusion Models
AI总结：
针对扩散模型生成一致可控视频难度大、成本高的问题，本文提出噪声结晶与液体噪声两种零样本方法，修改图像扩散模型的输入噪声生成视频，还拓展了相关应用并得出VAE嵌入的理论见解。
摘要：
尽管扩散模型在图像生成方面表现强大，但生成一致且可控的视频仍是扩散模型长期存在的难题。视频模型需要大量训练与计算资源，导致成本高昂且环境影响较大。此外，当前视频模型对输出运动的控制能力有限。本文提出一种新颖的视频生成方法，通过增强图像扩散模型来生成连续动画帧，同时保留精细细节。这些技术无需训练任何视频参数（零样本），只需修改潜在扩散模型中的输入噪声，即可应用于现有图像模型。文中提出两种互补方法：噪声结晶（Noise crystallization）可确保一致性，但由于潜在嵌入尺寸减小，仅适用于大尺度运动；液体噪声（Liquid noise）则以一致性为代价换取更高灵活性，且无分辨率限制。这些核心概念还可支持其他应用，如重新打光、无缝超分辨率和改进的视频风格迁移。此外，本文还对潜在扩散模型使用的VAE嵌入进行了探索，得出了颇具价值的理论见解，例如一种构建人类可解释潜在空间的方法。
英文摘要：
Although powerful for image generation, consistent and controllable video is a longstanding problem for diffusion models. Video models require extensive training and computational resources, leading to high costs and large environmental impacts. Moreover, video models currently offer limited control of the output motion. This paper introduces a novel approach to video generation by augmenting image diffusion models to create sequential animation frames while maintaining fine detail. These techniques can be applied to existing image models without training any video parameters (zero-shot) by altering the input noise in a latent diffusion model. Two complementary methods are presented. Noise crystallization ensures consistency but is limited to large movements due to reduced latent embedding sizes. Liquid noise trades consistency for greater flexibility without resolution limitations. The core concepts also allow other applications such as relighting, seamless upscaling, and improved video style transfer. Furthermore, an exploration of the VAE embedding used for latent diffusion models is performed, resulting in interesting theoretical insights such as a method for human-interpretable latent spaces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
