GriDiT: 基于网格的因子化扩散用于高效长图像序列生成
GriDiT: Factorized Grid-Based Diffusion for Efficient Long Image Sequence Generation
AI总结：
本文提出GriDiT方法，通过因子化生成长图像序列，先生成低分辨率粗序列再高分辨率细化，利用Diffusion Transformer的自注意力机制提升生成质量与效率。
摘要：
现代深度学习方法通常将图像序列视为连续堆叠帧的大型张量。然而，这种简单表示在当前最先进的方法中是否理想？本文在生成模型的背景下探讨此问题，旨在设计更有效的图像序列建模方法。观察到当前最先进的图像序列生成方法的效率和瓶颈，我们展示出通过将生成过程分解为先生成低分辨率粗序列再高分辨率细化单个帧，可以提高生成过程。我们训练一个仅基于子采样帧的网格图像的生成模型。然而，我们利用Diffusion Transformer（DiT）的强自注意力机制来捕捉帧间的相关性。实际上，我们的方法将2D图像生成器扩展为低分辨率3D图像序列生成器，而无需引入任何架构修改。随后，我们单独超分辨率每个帧以添加序列无关的高分辨率细节。这种方法提供了多种优势，并可以克服该领域中现有方法的关键限制。与现有图像序列生成模型相比，我们的方法在合成质量、序列间一致性、任意长度序列的高保真生成以及推理时间和训练数据使用效率方面均表现出色。此外，我们的简单方法使我们的方法能够有效地泛化到多样化的数据领域，这些领域通常需要额外的先验知识和监督来在生成背景下建模。我们的方法在多个数据集上均优于现有方法，在质量和推理速度（至少快两倍）方面表现优异。
英文摘要：
Modern deep learning methods typically treat image sequences as large tensors of sequentially stacked frames. However, is this straightforward representation ideal given the current state-of-the-art (SoTA)? In this work, we address this question in the context of generative models and aim to devise a more effective way of modeling image sequence data. Observing the inefficiencies and bottlenecks of current SoTA image sequence generation methods, we showcase that rather than working with large tensors, we can improve the generation process by factorizing it into first generating the coarse sequence at low resolution and then refining the individual frames at high resolution. We train a generative model solely on grid images comprising subsampled frames. Yet, we learn to generate image sequences, using the strong self-attention mechanism of the Diffusion Transformer (DiT) to capture correlations between frames. In effect, our formulation extends a 2D image generator to operate as a low-resolution 3D image-sequence generator without introducing any architectural modifications. Subsequently, we super-resolve each frame individually to add the sequence-independent high-resolution details. This approach offers several advantages and can overcome key limitations of the SoTA in this domain. Compared to existing image sequence generation models, our method achieves superior synthesis quality and improved coherence across sequences. It also delivers high-fidelity generation of arbitrary-length sequences and increased efficiency in inference time and training data usage. Furthermore, our straightforward formulation enables our method to generalize effectively across diverse data domains, which typically require additional priors and supervision to model in a generative context. Our method consistently outperforms SoTA in quality and inference speed (at least twice-as-fast) across datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
