Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining long-range attention over extended video streams, resulting in enhanced continuity and consistency. Consequently, a Stable Diffusion model fine-tuned with RAIN in just a few epochs can produce video streams in real-time and low latency without much compromise in quality or consistency, up to infinite long. Despite its advanced capabilities, the RAIN only introduces a few additional 1D attention blocks, imposing minimal additional burden. Experiments in benchmark datasets and generating super-long videos demonstrating that RAIN can animate characters in real-time with much better quality, accuracy, and consistency than competitors while costing less latency. All code and models will be made publicly available.
StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
State Key Laboratory of CAD&CG, Zhejiang University(浙江大学计算机辅助设计与图形学国家重点实验室);Ant Group(蚂蚁集团);College of Computer Science, Zhejiang University(浙江大学计算机学院)
RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Computer Science and Technology, Harbin University of Science and Technology(哈尔滨理工大学计算机科学与技术学院)
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Meituan New Laboratory of Pattern Recognition (NLPR), CASIA(美团模式识别新实验室(NLPR),中国科学院自动化所);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Meituan Division of AMC and Department of ECE, HKUST(美团AMC部门和香港科技大学电子与计算机工程系);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,中国科学院自动化所);Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences(美团人工智能学院,中国科学院大学);Meituan(美团)
CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving
Computer Science & Engineering Department, German University in Cairo (GUC), Egypt(德国开罗大学(GUC)计算机科学与工程系,埃及);C-DRiVeS Lab: Cognitive Driving Research in Vehicular Systems, Cairo, Egypt(认知驾驶系统实验室(C-DRiVeS Lab),开罗,埃及)
Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong Polytechnic University(香港理工大学)