arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04239cs.CV
收藏

DiverseDiT:扩散Transformer中的多样化表示学习

DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers

Fudan University(复旦大学);Shanghai Academy of AI for Science(上海人工智能科学研究院);Shanghai Innovation Institute(上海创新研究院)

Mengping Yang, Zhiyu Tan, Binglei Li, Xiaomeng Yang, Hesen Chen, Hao Li

AI总结:

通过分析扩散Transformer的表示动力学,发现块间表示多样性是关键,提出DiverseDiT框架,利用长残差连接和多样性损失促进多样特征学习,在ImageNet上提升性能并加速收敛。

摘要:

扩散Transformer(DiTs)的最新突破因其卓越的可扩展性而彻底改变了视觉合成领域。为了增强DiTs捕获有意义内部表示的能力,最近的工作如REPA引入了外部预训练编码器进行表示对齐。然而,DiTs内部表示学习的潜在机制尚不明确。为此,我们首先系统地研究了DiTs的表示动力学。通过分析不同设置下内部表示的演变和影响,我们揭示了块间的表示多样性是有效学习的关键因素。基于这一关键洞察,我们提出了DiverseDiT,一种明确促进表示多样性的新颖框架。DiverseDiT结合了长残差连接以多样化块间的输入表示,以及表示多样性损失以鼓励块学习不同的特征。在ImageNet 256x256和512x512上的大量实验表明,我们的DiverseDiT在应用于不同大小的不同骨干网络时,即使在具有挑战性的单步生成设置下,也能产生一致的性能提升和收敛加速。此外,我们展示了DiverseDiT与现有表示学习技术互补,从而带来进一步的性能提升。我们的工作为DiTs的表示学习动力学提供了宝贵的见解,并提供了增强其性能的实用方法。

英文摘要:

Recent breakthroughs in Diffusion Transformers (DiTs) have revolutionized the field of visual synthesis due to their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs are not well understood. To this end, we first systematically investigate the representation dynamics of DiTs. Through analyzing the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a crucial factor for effective learning. Based on this key insight, we propose DiverseDiT, a novel framework that explicitly promotes representation diversity. DiverseDiT incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet 256x256 and 512x512 demonstrate that our DiverseDiT yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes, even when tested on the challenging one-step generation setting. Furthermore, we show that DiverseDiT is complementary to existing representation learning techniques, leading to further performance gains. Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑