arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.07730cs.CVcs.AIcs.LGcs.MM
收藏

STIV:可扩展的文本与图像条件视频生成

STIV: Scalable Text and Image Conditioned Video Generation

Apple(苹果公司);University of California, Los Angeles(加州大学洛杉矶分校)
Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Lezhi Li, Yiz… 展开作者

Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Lezhi Li, Yizhou Sun, Kai-Wei Chang, Yinfei Yang

AI总结:

本研究提出可扩展的文本-图像条件视频生成方法STIV,通过帧替换与联合条件引导实现多任务支持,8.7B模型在VBench基准上取得超越主流模型的性能,为领域发展提供透明方案。

摘要:

视频生成领域已取得显著进展,但目前仍迫切需要一套清晰、系统的方案,来指导鲁棒且可扩展模型的开发。在本研究中,我们开展了一项全面研究,系统探索了模型架构、训练方案与数据策展策略之间的相互作用,最终提出了一种简单且可扩展的文本-图像条件视频生成方法,命名为STIV。我们的框架通过帧替换将图像条件整合到Diffusion Transformer(DiT,扩散Transformer)中,同时通过联合图像-文本条件的无分类器引导融入文本条件。这种设计使STIV能够同时执行文本生成视频(T2V)和文本-图像生成视频(TI2V)任务。此外,STIV可轻松扩展至多种应用场景,例如视频预测、帧插值、多视图生成和长视频生成等。通过在T2I、T2V和TI2V任务上开展的全面消融实验,STIV尽管设计简单,却展现出了强劲的性能。一个分辨率为512的8.7B参数模型在VBench T2V任务上取得了83.1分,超过了CogVideoX-5B、Pika、Kling和Gen-3等领先的开源与闭源模型。同等规模的模型在512分辨率下的VBench I2V任务上也取得了90.1分的当前最优结果。通过为构建前沿视频生成模型提供一套透明且可扩展的方案,我们旨在为未来研究提供支持,推动更通用、更可靠的视频生成解决方案的发展。

英文摘要:

The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑