扩散Transformer的自回归蒸馏
Autoregressive Distillation of Diffusion Transformers
AI总结：
针对扩散Transformer迭代采样资源消耗大及现有蒸馏方法暴露偏差问题，提出自回归蒸馏（ARD）利用ODE历史轨迹预测未来步骤，在ImageNet和T2I任务中实现低资源消耗、低FID退化及优提示一致性。
摘要：
采用Transformer架构的扩散模型在生成高保真图像和高分辨率可扩展性方面展现出良好性能，但合成所需的迭代采样过程资源消耗极大。现有研究聚焦于将概率流ODE的解决方案蒸馏为少步学生模型，然而这些方法依赖最新去噪样本作为输入，易受暴露偏差影响。为此，我们提出自回归蒸馏（ARD）方法，利用ODE的历史轨迹预测未来步骤，具有两大优势：1）通过使用不易积累误差的预测历史轨迹减轻暴露偏差；2）将ODE轨迹的历史信息作为更有效的粗粒度信息源。ARD对教师Transformer架构进行修改，添加逐token时间嵌入标记轨迹历史中的每个输入，并采用块级因果注意力掩码训练；仅在Transformer低层融入历史输入可提升性能与效率。在ImageNet类条件生成和文本到图像（T2I）合成任务中验证显示，ARD在ImageNet-256上仅增加1.1%额外FLOPs，FID退化减少5倍；4步采样FID达1.84，在1024p文本到图像任务中，与教师模型相比FID下降极小，且提示一致性得分优于公开蒸馏模型。项目页面：https://github.com/alsdudrla10/ARD。
英文摘要：
Diffusion models with transformer architectures have demonstrated promising capabilities in generating high-fidelity images and scalability for high resolution. However, iterative sampling process required for synthesis is very resource-intensive. A line of work has focused on distilling solutions to probability flow ODEs into few-step student models. Nevertheless, existing methods have been limited by their reliance on the most recent denoised samples as input, rendering them susceptible to exposure bias. To address this limitation, we propose AutoRegressive Distillation (ARD), a novel approach that leverages the historical trajectory of the ODE to predict future steps. ARD offers two key benefits: 1) it mitigates exposure bias by utilizing a predicted historical trajectory that is less susceptible to accumulated errors, and 2) it leverages the previous history of the ODE trajectory as a more effective source of coarse-grained information. ARD modifies the teacher transformer architecture by adding token-wise time embedding to mark each input from the trajectory history and employs a block-wise causal attention mask for training. Furthermore, incorporating historical inputs only in lower transformer layers enhances performance and efficiency. We validate the effectiveness of ARD in a class-conditioned generation on ImageNet and T2I synthesis. Our model achieves a $5\times$ reduction in FID degradation compared to the baseline methods while requiring only 1.1\% extra FLOPs on ImageNet-256. Moreover, ARD reaches FID of 1.84 on ImageNet-256 in merely 4 steps and outperforms the publicly available 1024p text-to-image distilled models in prompt adherence score with a minimal drop in FID compared to the teacher. Project page: https://github.com/alsdudrla10/ARD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
