arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.20404cs.CV
收藏

Open-Sora:为所有人普及高效视频生成

Open-Sora: Democratizing Efficient Video Production for All

HPC-AI Tech(HPC-AI 科技)

Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, Yang You

AI总结:

本文提出开源视频生成模型 Open-Sora,通过时空解耦的 STDiT 扩散框架和压缩 3D 自编码器,实现最长 15 秒、720p、任意宽高比的高效视频生成,并公开全部代码与权重以推动视觉智能普及。

摘要:

视觉和语言是人类的两种基础感官,它们共同构建了我们的认知能力与智能。尽管人工智能在语言能力方面已取得重大突破,但人工视觉智能,尤其是生成和模拟我们所看到世界的能力,仍远远落后。为了促进人工视觉智能的发展与普及,我们创建了 Open-Sora,这是一个开源的视频生成模型,旨在生成高保真视频内容。Open-Sora 支持广泛的视觉生成任务,包括文本到图像生成、文本到视频生成和图像到视频生成。该模型利用先进的深度学习架构以及训练/推理技术来实现灵活的视频合成,能够生成最长 15 秒、最高 720p 分辨率以及任意宽高比的视频内容。具体而言,我们提出了 Spatial-Temporal Diffusion Transformer(STDiT),这是一种高效的视频扩散框架,将空间注意力和时间注意力解耦。我们还引入了一个高度压缩的 3D 自编码器,使表示更加紧凑,并通过专门的训练策略进一步加速训练。通过这一举措,我们旨在促进 AI 内容创作社区中的创新、创造力和包容性。通过秉持开源原则,Open-Sora 使所有人都能完全获取所有训练/推理/数据准备代码以及模型权重。所有资源均公开提供于:https://github.com/hpcaitech/Open-Sora。

英文摘要:

Vision and language are the two foundational senses for humans, and they build up our cognitive ability and intelligence. While significant breakthroughs have been made in AI language ability, artificial visual intelligence, especially the ability to generate and simulate the world we see, is far lagging behind. To facilitate the development and accessibility of artificial visual intelligence, we created Open-Sora, an open-source video generation model designed to produce high-fidelity video content. Open-Sora supports a wide spectrum of visual generation tasks, including text-to-image generation, text-to-video generation, and image-to-video generation. The model leverages advanced deep learning architectures and training/inference techniques to enable flexible video synthesis, which could generate video content of up to 15 seconds, up to 720p resolution, and arbitrary aspect ratios. Specifically, we introduce Spatial-Temporal Diffusion Transformer (STDiT), an efficient diffusion framework for videos that decouples spatial and temporal attention. We also introduce a highly compressive 3D autoencoder to make representations compact and further accelerate training with an ad hoc training strategy. Through this initiative, we aim to foster innovation, creativity, and inclusivity within the community of AI content creation. By embracing the open-source principle, Open-Sora democratizes full access to all the training/inference/data preparation codes as well as model weights. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑