发表机构
Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对扩散Transformer编码器残差连接限制抽象表征的问题，提出移除残差并融合多深度特征的DDT-RFE，在视觉理解和生成任务上均优于DDT。
AI 中文摘要
扩散模型在生成图像的同时学习语义表示。在解耦扩散Transformer（DDT）中，条件编码器提供特征，引导速度解码器进行去噪。为了在所有噪声水平下实现有效的去噪，这些特征必须同时捕捉高层抽象结构和低层细节。然而，编码器中的跳跃/残差连接允许浅层特征绕过连续变换，这可能限制渐进抽象，或至少使得难以解耦不同抽象层次。我们提出DDT-RFE，它移除每个编码器块中自注意力和MLP操作周围的残差连接，同时保持训练稳定。为了保留抽象丢弃但解码器仍需要的信息，我们将输入补丁嵌入与中间和最终编码器特征融合，形成编码器输出。因此，解码器可以访问来自多个编码器深度的信息，而每个编码器块能够学习更抽象的表征。DDT-RFE在视觉理解任务上相比DDT取得了全面改进，包括图像分类、语义分割、目标发现和语义对应，同时使用更少的编码器块。它还在ImageNet上的图像生成中实现了更低的FID。
英文摘要
Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity decoder in denoising. To enable effective denoising at all noise levels, these features must capture both high-level abstract structures and low-level details. However, skip/residual connections in the encoder allow shallow features to bypass successive transformations, which may limit progressive abstraction, or at least make it difficult to disentangle different levels of abstraction. We propose DDT-RFE, which removes the residual connections around the Self-Attention and MLP operations in each encoder block while maintaining stable training. To retain the information that abstraction discards but that the decoder still needs, we fuse the input patch embedding with intermediate and final encoder features to form the encoder output. The decoder thus has access to information from multiple encoder depths, while each encoder block is able to learn more abstract representations. DDT-RFE achieves overall improvements over DDT across visual understanding tasks, including image classification, semantic segmentation, object discovery, and semantic correspondence, while using fewer encoder blocks. It also achieves a lower FID for image generation on ImageNet.
