Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
Netflix Eyeline Studios（NetflixEyeline Studios）；Netflix；Stony Brook University（斯通布罗克大学）；University of Maryland（马里兰大学）；Stanford University（斯坦福大学）
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, Ning Yu
Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise sampling. This is achieved by just a change in data: we pre-process training videos to yield structured noise. Consequently, our method is agnostic to diffusion model design, requiring no changes to model architectures or training pipelines. Specifically, we propose a novel noise warping algorithm, fast enough to run in real time, that replaces random temporal Gaussianity with correlated warped noise derived from optical flow fields, while preserving the spatial Gaussianity. The efficiency of our algorithm enables us to fine-tune modern video diffusion base models using warped noise with minimal overhead, and provide a one-stop solution for a wide range of user-friendly motion control: local object motion control, global camera movement control, and motion transfer. The harmonization between temporal coherence and spatial Gaussianity in our warped noise leads to effective motion control while maintaining per-frame pixel quality. Extensive experiments and user studies demonstrate the advantages of our method, making it a robust and scalable approach for controlling motion in video diffusion models. Video results are available on our webpage: https://eyeline-labs.github.io/Go-with-the-Flow. Source code and model checkpoints are available on GitHub: https://github.com/Eyeline-Labs/Go-with-the-Flow.
Controllable person image generation aims to produce realistic human images with desirable attributes such as a given pose, cloth textures, or hairstyles. However, the large spatial misalignment between source and target images makes the standard image-to-image translation architectures unsuitable for this task. Most state-of-the-art methods focus on alignment for global pose-transfer tasks. However, they fail to deal with region-specific texture-transfer tasks, especially for person images with complex textures. To solve this problem, we propose a novel Spatially-Adaptive Warped Normalization…
Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE
University of Edinburgh（爱丁堡大学）；University of Groningen（格罗宁根大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University College London（伦敦大学学院）
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（向量人工智能研究所）
We introduce layered controllable video generation, where we, without any supervision, decompose the initial frame of a video into foreground and background layers, with which the user can control the video generation process by simply manipulating the foreground mask. The key challenges are the unsupervised foreground-background separation, which is ambiguous, and ability to anticipate user manipulations with access to only raw video sequences. We address these challenges by proposing a two-stage learning procedure. In the first stage, with the rich set of losses and dynamic foreground size p…
We present a generative model for controllable person image synthesis,as shown in Figure , which can be applied to pose-guided person image synthesis, $i.e.$, converting the pose of a source person image to the target pose while preserving the texture of that source person image, and clothing-guided person image synthesis, $i.e.$, changing the clothing texture of a source person image to the desired clothing texture. By explicitly establishing the dense correspondence between the target pose and the source image, we can effectively address the misalignment introduced by pose tranfer and genera…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Tencent AI Lab（腾讯人工智能实验室）
The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip. In this paper, for the sake of both furthering this exploration and our own interest in a realistic application, we study image-to-video translation and particularly focus on the videos of facial expressions. This problem challenges the deep neural networks by another temporal dimension comparing to the image-to-image translation. Moreover, its single input image fails most existing video generation methods that re…
ReImagine: Rethinking Controllable High-Quality Human Video Generation via Image-First Synthesis
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Future Network of Intelligence Institute, CUHK-Shenzhen, China（CUHK-深圳智能网络研究院）
MotionClone: Training-Free Motion Cloning for Controllable Video Generation
University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；NVIDIA AI Technology Center（英伟达人工智能技术中心）；Tsinghua University（清华大学）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）；Shanghai AI Laboratory（上海人工智能实验室）；Avolution AI
Restereo: Diffusion stereo video generation and restoration
Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；VIA-Center Saarbücken（萨尔布吕肯VIA中心）；University of Cambridge（剑桥大学）；Google（谷歌）；Google DeepMind（谷歌DeepMind）
