Denoising Diffusion models have exhibited remarkable capabilities in image generation. However, generating high-quality samples requires a large number of iterations. Knowledge distillation for diffusion models is an effective method to address this limitation with a shortened sampling process but causes degraded generative quality. Based on our analysis with bias-variance decomposition and experimental observations, we attribute the degradation to the spatial fitting error occurring in the training of both the teacher and student model. Accordingly, we propose $\textbf{S}$patial $\textbf{F}$itting-$\textbf{E}$rror $\textbf{R}$eduction $\textbf{D}$istillation model ($\textbf{SFERD}$). SFERD utilizes attention guidance from the teacher model and a designed semantic gradient predictor to reduce the student's fitting error. Empirically, our proposed model facilitates high-quality sample generation in a few function evaluations. We achieve an FID of 5.31 on CIFAR-10 and 9.39 on ImageNet 64$\times$64 with only one step, outperforming existing diffusion methods. Our study provides a new perspective on diffusion distillation by highlighting the intrinsic denoising ability of models. Project link: \url{https://github.com/Sainzerjj/SFERD}.
Denoising Diffusion models have demonstrated their proficiency for generative sampling. However, generating good samples often requires many iterations. Consequently, techniques such as binary time-distillation (BTD) have been proposed to reduce the number of network calls for a fixed architecture. In this paper, we introduce TRAnsitive Closure Time-distillation (TRACT), a new method that extends BTD. For single step diffusion,TRACT improves FID by up to 2.4x on the same architecture, and achieves new single-step Denoising Diffusion Implicit Models (DDIM) state-of-the-art FID (7.4 for ImageNet…
Bridging Diffusion Pruning and Step Distillation with Teacher-Aligned Repair
School of Big Data and Artificial Intelligence（大数据与人工智能学院）；Guangdong University of Finance and Economics（广东财经大学）；School of Computer Science（计算机科学学院）；Shandong University（山东大学）；GDUFE-UCM Joint Institute（广东财经大学-UCM联合学院）
FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models
University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generators
Zhejiang University of Technology（浙江工业大学）；University of New South Wales (UNSW Sydney)（新南威尔士大学（悉尼））；University of Auckland（奥克兰大学）；Zhejiang University（浙江大学）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Bio and Brain Engineering（生物与脑工程系）；Kim Jaechul Graduate School of AI（金在哲人工智能研究生院）；Brain and Cognitive Sciences（脑与认知科学系）
TurboTalk: Progressive Distillation for One-Step Audio-Driven Talking Avatar Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Meituan（美团）；The Hong Kong Polytechnic University（香港理工大学）；CAIR, HKISI, CAS（中国科学院CAS HKISI CAIR）
From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；SGIT AI Lab, State Grid Corporation of China（国家电网公司SGIT人工智能实验室）；Zhejiang University of Technology（浙江工业大学）；Baidu（百度）
Technical University of Munich（慕尼黑工业大学）；University of Southern California（南加州大学）
Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the corrupted image. While this over-parameteriz…
Massachusetts Institute of Technology（麻省理工学院）；University of Oxford（牛津大学）；MIT-IBM Watson AI Lab, IBM Research（IBM研究院麻省理工学院-IBM沃森人工智能实验室）；Toyota Research Institute（丰田研究院）；Meta Reality Labs Research（Meta现实实验室）
本文揭示了 SDS 过度平滑的原因在于其高方差噪声采样，并提出在每个更新步骤中通过反演 DDIM 恢复更好的噪声近似，从而显著提升 3D 生成质量且无需额外训练。
Adversarial Concept Distillation for One-Step Diffusion Personalization
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；VCIP, CS, Nankai University（南开大学计算机学院视觉计算与智能感知实验室）；Program of Computer Science, City University of Hong Kong (Dongguan)（香港城市大学（东莞）计算机科学项目）；City University of Hong Kong（香港城市大学）
Stochastic Transition-Map Distillation for Fast Probabilistic Inference
Department of Aerospace Engineering, Georgia Institute of Technology（佐治亚理工学院航空航天工程系）；Department of Computer Science, Georgia Institute of Technology（佐治亚理工学院计算机科学系）；Department of Mathematics, Florida State University（佛罗里达州立大学数学系）
Tel-Aviv University（特拉维夫大学）；Facebook AI Research（脸书人工智能研究院）；École Normale Supérieure Paris-Saclay（巴黎-萨克雷高等师范学院）
Generative diffusion processes are an emerging and effective tool for image and speech generation. In the existing methods, the underline noise distribution of the diffusion process is Gaussian noise. However, fitting distributions with more degrees of freedom, could help the performance of such generative models. In this work, we investigate other types of noise distribution for the diffusion process. Specifically, we show that noise from Gamma distribution provides improved results for image and speech generation. Moreover, we show that using a mixture of Gaussian noise variables in the diff…
Adversarial Robustness Distillation (ARD) is a novel method to boost the robustness of small models. Unlike general adversarial training, its robust knowledge transfer can be less easily restricted by the model capacity. However, the teacher model that provides the robustness of knowledge does not always make correct predictions, interfering with the student's robust performances. Besides, in the previous ARD methods, the robustness comes entirely from one-to-one imitation, ignoring the relationship between examples. To this end, we propose a novel structured ARD method called Contrastive Rela…
BCDDM: Branch-Corrected Denoising Diffusion Model for Black Hole Image Generation
College of information science and engineering, Hunan Normal University（信息科学与工程学院，湖南师范大学）；Department of Physics, Institute of Interdisciplinary Studies, Key Laboratory of Low Dimensional Quantum Structures and Quantum Control of Ministry of Education, Synergetic Innovation Center for Quantum Effects and Applications, Hunan Normal University（物理系，跨学科研究所，教育部低维量子结构与量子控制重点实验室，量子效应与应用协同创新中心，湖南师范大学）；Center for Gravitation and Cosmology, College of Physical Science and Technology, Yangzhou University（引力与宇宙学中心，物理科学与技术学院，扬州大学）
