本文提出开源视频生成模型 Open-Sora,通过时空解耦的 STDiT 扩散框架和压缩 3D 自编码器,实现最长 15 秒、720p、任意宽高比的高效视频生成,并公开全部代码与权重以推动视觉智能普及。
摘要:
视觉和语言是人类的两种基础感官,它们共同构建了我们的认知能力与智能。尽管人工智能在语言能力方面已取得重大突破,但人工视觉智能,尤其是生成和模拟我们所看到世界的能力,仍远远落后。为了促进人工视觉智能的发展与普及,我们创建了 Open-Sora,这是一个开源的视频生成模型,旨在生成高保真视频内容。Open-Sora 支持广泛的视觉生成任务,包括文本到图像生成、文本到视频生成和图像到视频生成。该模型利用先进的深度学习架构以及训练/推理技术来实现灵活的视频合成,能够生成最长 15 秒、最高 720p 分辨率以及任意宽高比的视频内容。具体而言,我们提出了 Spatial-Temporal Diffusion Transformer(STDiT),这是一种高效的视频扩散框架,将空间注意力和时间注意力解耦。我们还引入了一个高度压缩的 3D 自编码器,使表示更加紧凑,并通过专门的训练策略进一步加速训练。通过这一举措,我们旨在促进 AI 内容创作社区中的创新、创造力和包容性。通过秉持开源原则,Open-Sora 使所有人都能完全获取所有训练/推理/数据准备代码以及模型权重。所有资源均公开提供于:https://github.com/hpcaitech/Open-Sora。
英文摘要:
Vision and language are the two foundational senses for humans, and they build up our cognitive ability and intelligence. While significant breakthroughs have been made in AI language ability, artificial visual intelligence, especially the ability to generate and simulate the world we see, is far lagging behind. To facilitate the development and accessibility of artificial visual intelligence, we created Open-Sora, an open-source video generation model designed to produce high-fidelity video content. Open-Sora supports a wide spectrum of visual generation tasks, including text-to-image generation, text-to-video generation, and image-to-video generation. The model leverages advanced deep learning architectures and training/inference techniques to enable flexible video synthesis, which could generate video content of up to 15 seconds, up to 720p resolution, and arbitrary aspect ratios. Specifically, we introduce Spatial-Temporal Diffusion Transformer (STDiT), an efficient diffusion framework for videos that decouples spatial and temporal attention. We also introduce a highly compressive 3D autoencoder to make representations compact and further accelerate training with an ad hoc training strategy. Through this initiative, we aim to foster innovation, creativity, and inclusivity within the community of AI content creation. By embracing the open-source principle, Open-Sora democratizes full access to all the training/inference/data preparation codes as well as model weights. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.
Sora Generates Videos with Stunning Geometrical Consistency
Nankai University(南开大学);ByteDance Inc(字节跳动公司);Wuhan University(武汉大学);Nankai International Advanced Research Institute (Shenzhen Futian)(南开大学国际高等研究院(深圳福田))
Simple Visual Artifact Detection in Sora-Generated Videos
Joshigakuin Senior High School(女子学院高级中学);National Institute of Advanced Industrial Science and Technology (AIST)(国立先进工业科学技术研究院);Visual Geometry Group, University of Oxford(牛津大学视觉几何组)
Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarking
Tsinghua University(清华大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);South China University of Technology(华南理工大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室);Singapore Management University(新加坡管理学院)
Mora: Enabling Generalist Video Generation via A Multi-Agent Framework
University of Notre Dame(圣母大学);Lehigh University(理海大学);LinkedIn Corporation(领英公司);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学);Peking University(北京大学);Microsoft Research(微软研究院)
Gender Bias in Text-to-Video Generation Models: A case study of Sora
Aligarh Muslim University(阿利加尔穆斯林大学);VIT Bhopal University(韦洛尔理工大学博帕尔分校);Nanyang Technological University(南洋理工大学);Technical University of Munich(慕尼黑工业大学);Imperial College London(帝国理工学院);Edinburgh Napier University(爱丁堡龙比亚大学)
本文通过分析 Sora 对性别中立及刻板印象提示生成的视频,揭示其将特定性别与行为和职业不当关联的偏见及训练数据根源。
Sora as a World Model? A Complete Survey on Text-to-Video Generation
University of Electronic Science and Technology of China(电子科技大学);Kyung Hee University(韩国庆熙大学);The Hong Kong Polytechnic University(香港理工大学);Harbin Institute of Technology Shenzhen(哈尔滨工业大学深圳学院);Nota Inc.(Nota公司);Tongji University(同济大学)
Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身智能研究所);Shanghai Innovation Institute(上海创新研究院);Shanghai Key Laboratory of Multimodal Embodied AI(上海市多模态具身人工智能重点实验室);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);The Chinese University of Hong Kong(香港中文大学);Central South University(中南大学);Fudan University(复旦大学)
Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
T2VAttack: Adversarial Attack on Text-to-Video Diffusion Models
Hangzhou Institute for Advanced Study, UCAS, school of Intelligent Science and Technology(杭州高等研究院,UCAS,智能科学与技术学院);State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences (UCAS)(中国科学院大学)
T2VEval: Benchmark Dataset and Objective Evaluation Method for T2V-generated Videos
School of Information and Communication Engineering, Communication University of China, No.1 East Street, Dingfuzhuang, Chaoyang District, Beijing, China(信息与通信工程学院,中国传媒大学);Beijing Academy of Artificial Intelligence, No.150 Chengfu Road, Haidian District, Beijing, China(北京人工智能研究院)