arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04527cs.MMcs.CVcs.GR
收藏

Echo-Infinity: 学习演化记忆用于实时无限视频生成

Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation

The Chinese University of Hong Kong(香港中文大学);Joy Future Academy, JD(京东探索研究院);The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua University(清华大学);The University of Hong Kong(香港大学);University of Science and Technology of China(中国科学技术大学)

Yuxuan Bian, Zeyue Xue, Songchun Zhang, Shiyi Zhang, Weiyang Jin, Yaowei Li, Junhao Zhuang, Haoran Li, Jie Huang, Haoyang Huang, Nan Duan, Qiang Xu

AI总结:

提出Echo-Infinity框架,通过可学习的演化记忆以恒定成本动态过滤、抽象和压缩任意长度历史,结合统一相对RoPE方案,首次实现24小时实时无限视频生成。

摘要:

我们提出了Echo Infinity,一个面向实时无限视频生成的自回归(AR)框架,它采用可学习的演化记忆,以恒定成本动态过滤、抽象和压缩任意长度的历史。现有方法主要使用预定义的KV缓存调度、固定比例启发式压缩或推理时的RoPE适配来管理记忆。这些设计由于有限的缓存窗口和忽略自回归生成噪声,不可避免地丢失历史信息并放大复合误差。受人类记忆巩固的启发,Echo-Infinity用可学习的记忆查询替代手工设计的记忆管理,这些查询通过注意力和门控机制在过去的帧从局部窗口中被驱逐时更新。查询与视频扩散变换器(DiTs)进行端到端优化,形成一种演化记忆,支持任意压缩比,计算量恒定且与视频长度无关。它们还充当可泛化的生成先验,即使仅使用优化后的初始状态也能提高质量。我们进一步引入了统一相对RoPE方案,它将锚定帧固定从id 0开始,并让最新帧的id在训练和推理过程中最多增长到DiTs预训练的最大时间RoPE id,从而将模型从有限的RoPE约束中解放出来,并缩小训练-测试的RoPE外推差距。在长视频和短视频生成中,Echo-Infinity达到了最先进的性能,并且据我们所知,首次展示了有前景的24小时(>130万帧)实时滚动生成,为无限视频生成提供了一条实用路径。

英文摘要:

We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite video generation that employs a learnable evolving memory to dynamically filter, abstract, and compress any-length history at constant cost. Existing methods mainly curate memory with predefined KV-cache schedules, fixed-ratio heuristic compression, or inference-time RoPE adaptation. These designs inevitably lose historical information and amplify compounding errors due to their limited cache window and ignorance of autoregressive generation noise. Inspired by human memory consolidation, Echo-Infinity replaces handcrafted memory curation with learnable Memory Query, which are updated by attention and a gating mechanism when past frames are evicted from the local window. The queries are optimized end-to-end with the video diffusion transformers (DiTs), forming an evolving memory that supports arbitrary compression ratios with constant computation independent of video length. They also act as a generalizable generation prior, improving quality even when only the optimized initial state is used. We further introduce Unified Relative RoPE Recipe, which anchors the sink frames to start from id 0 and lets the newest frame id grow at most to the DiTs' pretrained maximum temporal RoPE id throughout training and inference, freeing the model from the finite RoPE constraint and closing the train-test RoPE extrapolation gap. In long and short video generation, Echo-Infinity achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time, suggesting a practical path toward infinite video generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑