arXiv 2412.141692025-03-02cs.CV

无需向量量化的自回归视频生成

Autoregressive Video Generation without Vector Quantization

Beijing University of Posts and Telecommunications（北京邮电大学）；Key Laboratory of Intelligent Information Processing, ICT, CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian University of Technology（大连理工大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）

Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, Xinlong Wang

备注：: Accepted to ICLR 2025. Project page at https://github.com/baaivision/NOVA

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出无需向量量化的自回归视频模型NOVA，通过时间逐帧与空间逐集合预测结合双向建模，以0.6B参数实现高效率、高保真的视频生成及零样本应用。

摘要：

本文提出了一种能够高效实现自回归视频生成的新方法。我们提出将视频生成问题重新表述为时间上逐帧预测与空间上逐集合预测的非量化自回归建模。不同于先前自回归模型中的光栅扫描预测或扩散模型中固定长度token的联合分布建模，我们的方法保持了GPT风格模型的因果性以实现灵活的上下文能力，同时在单个帧内利用双向建模以提高效率。基于所提出的方法，我们训练了一个无需向量量化的新型视频自回归模型，称为NOVA。结果表明，即使模型容量小得多（即0.6B参数），NOVA在数据效率、推理速度、视觉保真度和视频流畅性方面也超越了先前的自回归视频模型。NOVA在文本到图像生成任务中也优于最先进的图像扩散模型，且训练成本显著更低。此外，NOVA在更长的视频时长上具有良好泛化能力，并在一个统一模型中实现了多样的零样本应用。代码和模型已在https://github.com/baaivision/NOVA公开。

英文摘要：

This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at https://github.com/baaivision/NOVA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。