循环自回归扩散：全局记忆与局部注意力相结合
Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention
AI总结：
研究针对超长视频生成中现有模型的不足，提出循环自回归扩散（RAD）框架，结合循环块与局部注意力，解决训练推理差距等问题，在相关数据集实验中展现出长视频生成的优越性。
摘要：
视频生成的最新进展已从短视频的双向模型转向超长视频生成的自回归模型。以前的模型通常使用滑动窗口注意力来限制推理成本，但缺乏有效的内存压缩和超出窗口大小的长期生成的检索，导致遗忘和时空不一致问题。为了在固定内存预算下增强历史信息的保留，我们在扩散变换器（DiT）模型中额外加入了时间循环神经网络（RNN）层。具体来说，我们发现每个DiT层注意力之后的LSTM层与其他先进的RNN模块（如测试时训练（TTT）和Mamba2）具有可比的性能。此外，现有的扩散-RNN方法由于训练-推理差距或窗口间缺乏重叠而经常遭受性能下降。为了解决这些限制，我们提出了一种新颖的循环自回归扩散（RAD）框架，它利用循环块进行内存更新和检索，并通过对重叠滑动窗口的全注意力保留局部细节，且没有训练和推理差距。在记忆迷宫和我的世界数据集上的实验证明了我们具有全局记忆和局部注意力的框架在长视频生成方面的优越性。
英文摘要：
Recent advancements in video generation has shifted from bidirectional models for short videos to autoregressive ones for ultra long video generation. Previous models, which usually use sliding window attention to restrict inference cost, lack effective memory compression and retrieval for long-term generation beyond the window size, leading to issues of forgetting and spatiotemporal inconsistencies. To enhance the retention of historical information with a fixed memory budget, we additionally incorporate temporal recurrent neural network (RNN) layers into the diffusion transformer (DiT) model. Specifically, we found that a LSTM layer after attention at each DiT layer achieves comparable performance to other state-of-the-art RNN blocks, such as Test-Time Training (TTT) and Mamba2. Moreover, existing diffusion-RNN approaches often suffer from performance degradation due to training-inference gap or the lack of overlap across windows. To address these limitations, we propose a novel Recurrent Autoregressive Diffusion (RAD) framework, which leverages recurrent blocks for memory update and retrieval and preserves local details by full attention on overlapping sliding windows, with no training and inference gap. Experiments on Memory Maze and Minecraft datasets demonstrate the superiority for long video generation by our framework with global memory and local attention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
