LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity
Princeton University（普林斯顿大学）；Meta
Hongjie Wang, Chih-Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, Peter Vajda, Niraj K. Jha, Xiaoliang Dai
文本到视频生成增强了内容创作，但计算密集度极高：扩散变换器（DiTs）的计算成本随像素数量呈二次方增长。这使得分钟级视频生成极为昂贵，限制了大多数现有模型仅能生成10-20秒长度的视频。我们提出了一种线性复杂度的文本到视频生成（LinGen）框架，其成本随像素数量线性增长。LinGen首次实现了在单个GPU上生成高分辨率分钟级视频，且不牺牲质量。它用称为MATE的线性复杂度块替换了计算主导且具有二次复杂度的自注意力块，MATE由MA分支和TE分支组成。MA分支针对短到长距离相关性，结合了双向Mamba2块、我们的令牌重排方法Rotary Major Scan以及为长视频生成开发的回顾令牌。TE分支是一种新颖的时间Swin注意力块，专注于相邻令牌和中距离令牌之间的时间相关性。MATE块解决了Mamba的邻接保持问题，显著提高了生成视频的一致性。实验结果表明，LinGen在视频质量上优于DiT（胜率75.6%），同时计算量（延迟）最多减少15倍（11.5倍）。此外，自动指标和人工评估均表明，我们的LinGen-4B生成的视频质量与最先进模型相当（相对于Gen-3、LumaLabs和Kling的胜率分别为50.5%、52.1%和49.1%）。这为小时级电影生成和实时交互式视频生成铺平了道路。我们在项目网站上提供了68秒视频生成结果和更多示例：https://lineargen.github.io/。
英文摘要：
Text-to-video generation enhances content creation but is highly computationally intensive: The computational cost of Diffusion Transformers (DiTs) scales quadratically in the number of pixels. This makes minute-length video generation extremely expensive, limiting most existing models to generating videos of only 10-20 seconds length. We propose a Linear-complexity text-to-video Generation (LinGen) framework whose cost scales linearly in the number of pixels. For the first time, LinGen enables high-resolution minute-length video generation on a single GPU without compromising quality. It replaces the computationally-dominant and quadratic-complexity block, self-attention, with a linear-complexity block called MATE, which consists of an MA-branch and a TE-branch. The MA-branch targets short-to-long-range correlations, combining a bidirectional Mamba2 block with our token rearrangement method, Rotary Major Scan, and our review tokens developed for long video generation. The TE-branch is a novel TEmporal Swin Attention block that focuses on temporal correlations between adjacent tokens and medium-range tokens. The MATE block addresses the adjacency preservation issue of Mamba and improves the consistency of generated videos significantly. Experimental results show that LinGen outperforms DiT (with a 75.6% win rate) in video quality with up to 15$\times$ (11.5$\times$) FLOPs (latency) reduction. Furthermore, both automatic metrics and human evaluation demonstrate our LinGen-4B yields comparable video quality to state-of-the-art models (with a 50.5%, 52.1%, 49.1% win rate with respect to Gen-3, LumaLabs, and Kling, respectively). This paves the way to hour-length movie generation and real-time interactive video generation. We provide 68s video generation results and more examples in our project website: https://lineargen.github.io/.
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Ring Forcing: Towards Precise Long-Term Memory for Autoregressive Video Diffusion
Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）；Peking University（北京大学）；University of California, Berkeley（加州大学伯克利分校）；ByteDance（字节跳动）
SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models
Huazhong University of Science and Technology（华中科技大学）；Zhongguancun Academy（中关村学院）；Tsinghua University（清华大学）；Modelbest；Peking University（北京大学）；Renmin University of China（中国人民大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers
Dept. of Comp. Sci. \& Tech., BNRist Center, THU-Bosch ML Center, Tsinghua University.；The University of Texas at Austin.；Gaoling School of Artificial Intelligence Renmin University of China Beijing, China.；Beijing Key Laboratory of Research on Large Models；Engineering Research Center of Next-Generation Intelligent Search；Pazhou Laboratory (Huangpu)
