DIVER: 通过表达性语义恢复深入挖掘蒸馏数据
DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery
AI总结：
提出双阶段蒸馏框架DIVER，利用预训练扩散模型通过语义继承、引导和融合恢复蒸馏数据的表达性语义，提升跨架构泛化能力。
摘要：
数据集蒸馏旨在从原始数据集中合成一个紧凑的代理数据集，该数据集不可读或非原始，以保护隐私并实现高效学习。然而，先前的方法通常采用单阶段蒸馏范式，该范式会学习过度适应先验架构的特定模式，从而抑制语义表达并导致跨异构架构的性能下降。为了解决这个问题，我们提出了一种新颖的双阶段蒸馏框架，称为${ extbf{DIVER}}$，它利用预训练的扩散模型通过表达性语义恢复深入挖掘蒸馏数据，整个过程包括语义继承、引导和融合。语义继承将抽象蒸馏图像的高级语义蒸馏到潜在空间中，以过滤掉架构特定的“噪声”并保留内在语义。此外，语义引导通过指导反向过程来改善原始语义的保留。最后，语义融合被设计为仅在反向过程的具体阶段提供语义引导，防止语义模糊和伪影，同时保持引导信息。大量实验验证了DIVER在改进经典蒸馏技术和显著提升跨架构泛化方面的有效性和效率，在ImageNet（256×256）上仅需与原始DiT相当的处理时间，且仅使用4 GB GPU内存。
英文摘要：
Dataset distillation aims to synthesize a compact proxy dataset that is unreadable or non-raw from the original dataset for privacy protection and highly efficient learning. However, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning specific patterns that overfit on a prior architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this issue, we propose a novel dual-stage distillation framework called ${\textbf{DIVER}}$, which leverages the pre-trained diffusion model to dive deeper into $\textbf{DI}$stilled data $\textbf{V}$ia $\textbf{E}$xpressive semantic $\textbf{R}$ecovery, an entire process of semantic inheritance, guidance, and fusion. Semantic inheritance distills high-level semantics of abstract distilled images into the latent space to filter out architecture-specific ``noise" and retain the intrinsic semantics. Furthermore, semantic guidance improves the preservation of the original semantics by directing the reverse procedure. Finally, semantic fusion is designed to provide semantic guidance only during the concrete phase of the reverse process, preventing semantic ambiguity and artifacts while maintaining the guidance information. Extensive experiments validate the effectiveness and efficiency of DIVER in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256$\times$256) with only 4 GB of GPU memory usage. Code is available: https://github.com/einsteinxia/DIVER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
