arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02371cs.CV
收藏

OpenVid-1M:用于文本到视频生成的大规模高质量数据集

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

Nanjing University(南京大学);ByteDance(字节跳动);Nankai University(南开大学)

Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, Ying Tai

AI总结:

针对文本到视频生成缺乏精确高质量开源数据集和未充分利用文本信息的问题,本文构建含百万级高质量文本-视频对的OpenVid-1M数据集及高清子集,并提出多模态视频扩散Transformer,提升生成效果。

摘要:

得益于大型多模态模型Sora,文本到视频(T2V)生成最近获得了广泛关注。然而,T2V生成仍面临两个重要挑战:1)缺乏精确的开源高质量数据集。之前流行的视频数据集(如WebVid-10M和Panda-70M)要么质量低,要么对大多数研究机构来说太大。因此,收集用于T2V生成的精确高质量文本-视频对具有挑战性但至关重要。2)忽视充分利用文本信息。最近的T2V方法专注于视觉Transformer,使用简单的交叉注意力模块进行视频生成,未能从文本提示中彻底提取语义信息。为解决这些问题,我们引入OpenVid-1M,一个具有丰富描述的精确高质量数据集。该开放场景数据集包含超过100万对文本-视频对,促进T2V生成研究。此外,我们从OpenVid-1M中精选43.3万条1080p视频创建OpenVidHD-0.4M,推动高清视频生成。我们还提出一种新颖的多模态视频扩散Transformer(MVDiT),能够挖掘视觉标记的结构信息和文本标记的语义信息。大量实验和消融研究验证了OpenVid-1M相对于现有数据集的优越性以及MVDiT的有效性。

英文摘要:

Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges: 1) Lacking a precise open sourced high-quality dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M, are either with low quality or too large for most research institutions. Therefore, it is challenging but crucial to collect a precise high-quality text-video pairs for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of thoroughly extracting semantic information from text prompt. To address these issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive captions. This open-scenario dataset contains over 1 million text-video pairs, facilitating research on T2V generation. Furthermore, we curate 433K 1080p videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition video generation. Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens. Extensive experiments and ablation studies verify the superiority of OpenVid-1M over previous datasets and the effectiveness of our MVDiT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑