基于下一帧预测的长上下文自回归视频建模
Long-Context Autoregressive Video Modeling with Next-Frame Prediction
AI总结：
针对长视频自回归建模计算成本高的问题，提出基于非对称分块核的长短时上下文建模方法，在长短视频生成上均达SOTA，提供了有效基线。
摘要：
长上下文视频建模对于让生成式模型发挥世界模拟器的功能至关重要，因为这类模型必须在较长时间跨度内保持时间连贯性。然而，现有大多数模型都在短视频片段上训练，即便在测试时进行外推，其捕捉长程依赖关系的能力也受到限制。虽然直接在长视频上训练是一种自然而然的解决方案，但视觉token的快速增长导致该方案计算成本过高，难以实现。\n为支持高效长上下文视频建模的探索，我们首先构建了一个性能强劲的自回归基线模型Frame AutoRegressive（FAR，帧自回归模型）。FAR对连续帧之间的时间依赖关系进行建模，收敛速度快于视频扩散Transformer，且性能优于token级自回归模型。基于该基线，我们发现视频自回归过程中存在上下文冗余：邻近帧对维持时间一致性至关重要，而较远帧主要充当上下文记忆。为消除这种冗余，我们提出采用非对称patchify（分块）核的长短时上下文建模方法：对较远帧使用大核以减少冗余token，对局部帧使用标准核以保留细粒度细节。该方法显著降低了长视频的训练成本。我们的方法在短视频和长视频生成任务上均取得了当前最优结果，为长上下文自回归视频建模提供了有效的基线。
英文摘要：
Long-context video modeling is essential for enabling generative models to function as world simulators, as they must maintain temporal coherence over extended time spans. However, most existing models are trained on short clips, limiting their ability to capture long-range dependencies, even with test-time extrapolation. While training directly on long videos is a natural solution, the rapid growth of vision tokens makes it computationally prohibitive. To support exploring efficient long-context video modeling, we first establish a strong autoregressive baseline called Frame AutoRegressive (FAR). FAR models temporal dependencies between continuous frames, converges faster than video diffusion transformers, and outperforms token-level autoregressive models. Based on this baseline, we observe context redundancy in video autoregression. Nearby frames are critical for maintaining temporal consistency, whereas distant frames primarily serve as context memory. To eliminate this redundancy, we propose the long short-term context modeling using asymmetric patchify kernels, which apply large kernels to distant frames to reduce redundant tokens, and standard kernels to local frames to preserve fine-grained detail. This significantly reduces the training cost of long videos. Our method achieves state-of-the-art results on both short and long video generation, providing an effective baseline for long-context autoregressive video modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
