学习可堆叠和可跳过的LEGO模块以实现高效、可重构和可变分辨率的扩散建模
Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling
AI总结:
本研究提出LEGO模块,通过集成局部特征增强和全局内容编排构建可重构扩散主干,支持选择性跳过模块以降低采样成本并实现可变分辨率生成,在保持生成性能的同时显著提升效率。
摘要:
扩散模型在生成逼真图像方面表现出色,但在训练和采样中均存在显著的计算成本。尽管已有多种技术应对这些计算挑战,但设计一个高效且适应性强的网络主干用于迭代细化仍是一个较少被探索的问题。当前的选择如U-Net和Vision Transformer通常依赖资源密集的深度网络,且缺乏以可变分辨率或使用比训练时更小的网络生成图像所需的灵活性。本研究引入了LEGO模块,它无缝集成了局部特征增强和全局内容编排。这些模块可以堆叠以创建一个测试时可重构的扩散主干,允许选择性跳过模块以降低采样成本,并生成比训练数据分辨率更高的图像。LEGO模块通过MLP丰富局部区域,并使用Transformer块对其进行变换,同时在所有模块中保持一致的全分辨率图像。实验结果表明,LEGO模块提高了训练效率,加速了收敛,并促进了可变分辨率图像生成,同时保持了强大的生成性能。此外,与其他方法相比,LEGO显著减少了采样时间,使其成为扩散模型的重要增强。我们的代码和项目页面可在https://jegzheng.github.io/LEGODiffusion获取。
英文摘要:
Diffusion models excel at generating photo-realistic images but come with significant computational costs in both training and sampling. While various techniques address these computational challenges, a less-explored issue is designing an efficient and adaptable network backbone for iterative refinement. Current options like U-Net and Vision Transformer often rely on resource-intensive deep networks and lack the flexibility needed for generating images at variable resolutions or with a smaller network than used in training. This study introduces LEGO bricks, which seamlessly integrate Local-feature Enrichment and Global-content Orchestration. These bricks can be stacked to create a test-time reconfigurable diffusion backbone, allowing selective skipping of bricks to reduce sampling costs and generate higher-resolution images than the training data. LEGO bricks enrich local regions with an MLP and transform them using a Transformer block while maintaining a consistent full-resolution image across all bricks. Experimental results demonstrate that LEGO bricks enhance training efficiency, expedite convergence, and facilitate variable-resolution image generation while maintaining strong generative performance. Moreover, LEGO significantly reduces sampling time compared to other methods, establishing it as a valuable enhancement for diffusion models. Our code and project page are available at https://jegzheng.github.io/LEGODiffusion.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。