Lumina-T2X:通过基于流的大型扩散 Transformer 将文本转换为任意模态、分辨率和时长
Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
AI总结:
提出 Lumina-T2X 统一框架,利用基于流的大型扩散 Transformer 和时空 token 化实现文本到多模态、任意分辨率及时长的生成,并以更低训练成本支持高分辨率图像与视频。
摘要:
Sora 揭示了扩展 Diffusion Transformer 以在任意分辨率、宽高比和时长下生成照片级真实图像和视频的潜力,但其仍缺乏足够的实现细节。在本技术报告中,我们介绍 Lumina-T2X 系列——一系列配备零初始化注意力的基于流的大型扩散 Transformer(Flag-DiT),作为一个统一框架,旨在根据文本指令将噪声转换为图像、视频、多视图 3D 对象和音频片段。通过对潜在时空空间进行分词,并引入 [nextline] 和 [nextframe] token 等可学习占位符,Lumina-T2X 无缝统一了不同模态在各种时空分辨率下的表示。这种统一方法支持在单一框架内针对不同模态进行训练,并允许在推理过程中灵活生成任意分辨率、宽高比和长度的多模态数据。RoPE、RMSNorm 和 flow matching 等先进技术增强了 Flag-DiT 的稳定性、灵活性和可扩展性,使 Lumina-T2X 的模型能够扩展至 70 亿参数,并将上下文窗口扩展至 128K token。这尤其有利于使用我们的 Lumina-T2I 模型创建超高清图像,以及使用我们的 Lumina-T2V 模型生成长的 720p 视频。值得注意的是,由 50 亿参数 Flag-DiT 驱动的 Lumina-T2I 仅需 6 亿参数朴素 DiT 的 35% 训练计算成本。我们进一步的综合分析强调了 Lumina-T2X 在分辨率外推、高分辨率编辑、生成一致 3D 视图以及合成具有无缝过渡视频方面的初步能力。我们期望 Lumina-T2X 的开源将进一步促进生成式 AI 社区的创造力、透明度和多样性。
英文摘要:
Sora unveils the potential of scaling Diffusion Transformer for generating photorealistic images and videos at arbitrary resolutions, aspect ratios, and durations, yet it still lacks sufficient implementation details. In this technical report, we introduce the Lumina-T2X family - a series of Flow-based Large Diffusion Transformers (Flag-DiT) equipped with zero-initialized attention, as a unified framework designed to transform noise into images, videos, multi-view 3D objects, and audio clips conditioned on text instructions. By tokenizing the latent spatial-temporal space and incorporating learnable placeholders such as [nextline] and [nextframe] tokens, Lumina-T2X seamlessly unifies the representations of different modalities across various spatial-temporal resolutions. This unified approach enables training within a single framework for different modalities and allows for flexible generation of multimodal data at any resolution, aspect ratio, and length during inference. Advanced techniques like RoPE, RMSNorm, and flow matching enhance the stability, flexibility, and scalability of Flag-DiT, enabling models of Lumina-T2X to scale up to 7 billion parameters and extend the context window to 128K tokens. This is particularly beneficial for creating ultra-high-definition images with our Lumina-T2I model and long 720p videos with our Lumina-T2V model. Remarkably, Lumina-T2I, powered by a 5-billion-parameter Flag-DiT, requires only 35% of the training computational costs of a 600-million-parameter naive DiT. Our further comprehensive analysis underscores Lumina-T2X's preliminary capability in resolution extrapolation, high-resolution editing, generating consistent 3D views, and synthesizing videos with seamless transitions. We expect that the open-sourcing of Lumina-T2X will further foster creativity, transparency, and diversity in the generative AI community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。