arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.19489cs.CV
收藏

RAIN:无限视频流的实时动画生成

RAIN: Real-time Animation of Infinite Video Stream

Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha

AI总结:

RAIN 提出一种基于帧-令牌高效注意力计算的实时视频流动画方案,可在单块 RTX 4090 上以低延迟生成无限长、高质量且连贯的视频流。

摘要:

实时动画在提升在线互动方面已获得极高人气,然而利用扩散模型实现高质量、实时且稳定的动画仍然具有挑战性,尤其是在消费级 GPU 上。现有方法难以高效生成长时间、连贯的视频流,通常受到延迟问题以及长时间生成后视觉质量下降的限制。本文提出 RAIN,一种能够在单块 RTX 4090 GPU 上以低延迟实时生成无限视频流的流水线解决方案。RAIN 的核心思想是在不同噪声水平和长时间间隔上高效计算帧-令牌注意力,同时并行去噪数量远多于以往基于流的方法的帧-令牌。该设计使 RAIN 能够以更短的延迟和更快的速度生成视频帧,同时保持对长视频流的远程注意力,从而增强连续性和一致性。因此,仅经过少量轮次微调并采用 RAIN 的 Stable Diffusion 模型即可实时、低延迟地生成视频流,且在质量和一致性上没有明显妥协,甚至可无限延长。尽管能力先进,RAIN 仅引入少量额外的一维注意力模块,增加的额外负担极小。在基准数据集上的实验以及超长视频生成结果表明,RAIN 能够以更低的延迟实时驱动角色动画,在质量、准确性和一致性上均优于竞争对手。所有代码和模型将公开发布。

英文摘要:

Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining long-range attention over extended video streams, resulting in enhanced continuity and consistency. Consequently, a Stable Diffusion model fine-tuned with RAIN in just a few epochs can produce video streams in real-time and low latency without much compromise in quality or consistency, up to infinite long. Despite its advanced capabilities, the RAIN only introduces a few additional 1D attention blocks, imposing minimal additional burden. Experiments in benchmark datasets and generating super-long videos demonstrating that RAIN can animate characters in real-time with much better quality, accuracy, and consistency than competitors while costing less latency. All code and models will be made publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑