层与时间步自适应可微令牌压缩比率用于高效扩散Transformer
Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers
AI总结：
提出DiffCR，通过可微压缩比率在令牌、层和时间步上动态路由计算，提升扩散Transformer效率，实现质量与效率的优越权衡。
摘要：
扩散Transformer（DiTs）在图像生成质量上已达到最先进水平，但存在高延迟和内存效率低下的问题，使其难以部署在资源受限设备上。一个主要的效率瓶颈是现有DiTs对图像的所有区域应用相同的计算量。然而，并非所有图像令牌都同等重要，某些局部区域需要更多计算，例如物体。为解决此问题，我们提出DiffCR，一种具有可微压缩比率的动态DiT推理框架，它自动学习在层和时间步上为每个图像令牌动态路由计算，从而实现高效的DiTs。具体而言，DiffCR集成了三个特性：（1）令牌级路由方案，每个DiT层包含一个路由器，与模型权重联合微调以预测令牌重要性分数。这样，不重要的令牌绕过整个层的计算；（2）层间可微比率机制，不同DiT层从零初始化自动学习不同的压缩比率，导致冗余层具有较大的压缩比率，而其他层保持较少压缩甚至不压缩；（3）时间步可微比率机制，每个去噪时间步学习其自身的压缩比率。所得模式显示，噪声较大的时间步具有较高比率，而图像变得清晰时比率较低。在文本到图像和图像修复任务上的大量实验表明，DiffCR有效捕捉了令牌、层和时间步轴上的动态性，与先前工作相比，在生成质量和效率之间实现了优越的权衡。项目网站可在https://www.haoranyou.com/diffcr获取。
英文摘要：
Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency, making them difficult to deploy on resource-constrained devices. One major efficiency bottleneck is that existing DiTs apply equal computation across all regions of an image. However, not all image tokens are equally important, and certain localized areas require more computation, such as objects. To address this, we propose DiffCR, a dynamic DiT inference framework with differentiable compression ratios, which automatically learns to dynamically route computation across layers and timesteps for each image token, resulting in efficient DiTs. Specifically, DiffCR integrates three features: (1) A token-level routing scheme where each DiT layer includes a router that is fine-tuned jointly with model weights to predict token importance scores. In this way, unimportant tokens bypass the entire layer's computation; (2) A layer-wise differentiable ratio mechanism where different DiT layers automatically learn varying compression ratios from a zero initialization, resulting in large compression ratios in redundant layers while others remain less compressed or even uncompressed; (3) A timestep-wise differentiable ratio mechanism where each denoising timestep learns its own compression ratio. The resulting pattern shows higher ratios for noisier timesteps and lower ratios as the image becomes clearer. Extensive experiments on text-to-image and inpainting tasks show that DiffCR effectively captures dynamism across token, layer, and timestep axes, achieving superior trade-offs between generation quality and efficiency compared to prior works. The project website is available at https://www.haoranyou.com/diffcr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
