Dataset distillation compresses the original data into compact synthetic datasets, reducing training time and storage while retaining model performance, enabling deployment under limited resources. Although recent decoupling-based distillation methods enable dataset distillation at large scale, they continue to face an efficiency gap: optimization-based decoupling methods achieve higher accuracy but demand intensive computation, whereas optimization-free decoupling methods are efficient but sacrifice accuracy. To overcome this trade-off, we propose Exploration--Exploitation Distillation (E$^2$D), a simple, practical method that minimizes redundant computation through an efficient pipeline that begins with full-image initialization to preserve semantic integrity and feature diversity. It then uses a two-phase optimization strategy: an exploration phase that performs uniform updates and identifies high-loss regions, and an exploitation phase that focuses updates on these regions to accelerate convergence. We evaluate E$^2$D on large-scale benchmarks, surpassing the state-of-the-art on ImageNet-1K while being $18\times$ faster, and on ImageNet-21K, our method substantially improves accuracy while remaining $4.3\times$ faster. These results demonstrate that targeted, redundancy-reducing updates, rather than brute-force optimization, bridge the gap between accuracy and efficiency in large-scale dataset distillation. Code is available at https://github.com/ncsu-dk-lab/E2D.
Accelerating Dataset Distillation via Model Augmentation
Zhejiang University（浙江大学）；Texas A&M University（德州农工大学）；Microsoft Research（微软研究院）；New York University（纽约大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；University of Connecticut（康涅狄格大学）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）；North Carolina State University（北卡罗来纳州立大学）
Catch-Up Distillation: You Only Need to Train Once for Accelerating Sampling
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Southeast University（东南大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
University of Maryland（马里兰大学）；Georgia Institute of Technology（佐治亚理工学院）；Honda Research Institute（本田研究院）
There is an increasing interest on accelerating neural networks for real-time applications. We study the student-teacher strategy, in which a small and fast student network is trained with the auxiliary information learned from a large and accurate teacher network. We propose to use conditional adversarial networks to learn the loss function to transfer knowledge from teacher to student. The proposed method is particularly effective for relatively small student networks. Moreover, experimental results show the effect of network size when the modern networks are used as student. We empirically …
Chongqing University（重庆大学）；San Diego State University（圣地亚哥州立大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Multi-Label Image Classification (MLIC) approaches usually exploit label correlations to achieve good performance. However, emphasizing correlation like co-occurrence may overlook discriminative features of the target itself and lead to model overfitting, thus undermining the performance. In this study, we propose a generic framework named Parallel Self-Distillation (PSD) for boosting MLIC models. PSD decomposes the original MLIC task into several simpler MLIC sub-tasks via two elaborated complementary task decomposition strategies named Co-occurrence Graph Partition (CGP) and Dis-occurrence G…
Aiming towards human-level generalization, there is a need to explore adaptable representation learning methods with greater transferability. Most existing approaches independently address task-transferability and cross-domain adaptation, resulting in limited generalization. In this paper, we propose UM-Adapt - a unified framework to effectively perform unsupervised domain adaptation for spatially-structured prediction tasks, simultaneously maintaining a balanced performance across individual tasks in a multi-task setting. To realize this, we propose two novel regularization strategies; a) Con…
Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
Digital Environment Research Institute (DERI), Queen Mary University of London（女王玛丽大学数字环境研究所）；School of Electronic Engineering and Computer Science, Queen Mary University of London（女王玛丽大学电子工程与计算机科学学院）
Elucidating the Preconditioning in Consistency Distillation
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab（计算机科学与技术系，人工智能研究所，BNRist中心，THBI实验室）；Tsinghua-Bosch Joint ML Center, Tsinghua University（清华大学-博世联合机器学习中心，清华大学）；Shengshu Technology（盛舒科技）；Pazhou Lab (Huangpu), Guangzhou, China（黄埔实验室（广州），中国）
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（信息科技学院，科学大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
SafeDivertor: Faithful Divertor Heat Flux Reconstruction from Macroscopic Plasma State Signals via Time-Frequency Prior Exploitation
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Institute of Plasma Physics, Chinese Academy of Sciences（中国科学院等离子体物理研究所）
Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of $N$-step teacher sampler with $N/2$-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed \textbf{C}lassifier-based \textbf{F}eature \textbf{D}istillation (CFD). Instead of aligning output images, we distill teacher's …
Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Peking University（北京大学）；Alibaba Group（阿里巴巴集团）
Learn Faster and Remember More: Balancing Exploration and Exploitation for Continual Test-time Adaptation
School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（电子、电气与通信工程学院，中国科学院大学）；School of Computer Science and Technology, University of Chinese Academy of Sciences（计算机科学与技术学院，中国科学院大学）；Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所）
Continuous-Time Distribution Matching for Few-Step Diffusion Distillation
VCIP, College of Computer Science, Nankai University（南开大学计算机科学学院VCIP）；Alibaba Group（阿里巴巴集团）；College of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
