基于下一片段扩散的Video-GPT
Video-GPT via Next Clip Diffusion
AI总结：
针对语言序列难以描述视觉时空细节的问题，提出将视频作为视觉建模新语言的Video-GPT，采用下一片段扩散范式预训练，在视频预测任务上达SOTA，且能适配6项主流视频任务，泛化性强。
摘要：
GPT已在自然语言处理领域展现出卓越成效，但语言序列不足以描述视觉世界中的时空细节，而视频序列则擅长捕捉这类细节。受此启发，本文提出一种简洁的Video-GPT，将视频作为视觉世界建模的新型语言。类比GPT中的下一词元预测任务，我们提出一种新颖的下一片段扩散范式用于Video-GPT的预训练。与现有工作不同，该独特范式可让Video-GPT根据历史中的干净片段对含噪片段进行自回归去噪，从而同时处理短期生成与长期预测任务。大量实验表明，本文的Video-GPT在视频预测任务上取得了当前最优性能，这是迈向世界建模的关键因素（Physics-IQ Benchmark：Video-GPT 34.97，对比Kling 23.64、Wan 20.89）。此外，该模型能很好地适配视频生成与理解领域的6项主流视频任务，展现出极强的下游泛化能力。项目页面地址为：https://zhuangshaobin.github.io/Video-GPT.github.io/。
英文摘要：
GPT has shown its remarkable success in natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such details. Motivated by this fact, we propose a concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-art performance on video prediction, which is the key factor towards world modeling (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89). Moreover, it can be well adapted on 6 mainstream video tasks in both video generation and understanding, showing its great generalization capacity in downstream. The project page is at https://zhuangshaobin.github.io/Video-GPT.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
