直接以最小信息损失压缩大规模数据集
Condensing Large-Scale Datasets Directly with Minimal Information Loss
AI总结：
针对现有数据集蒸馏方法因隐式双压缩导致信息损失和跨架构泛化差的问题，提出CIM框架，通过直接量化并最小化原始与合成数据集间的信息差距，实现高保真信息压缩，在ImageNet-1K上以IPC=10、80分钟训练达到48.7% Top-1准确率。
摘要：
最近在扩展数据集蒸馏方面的进展严重依赖于解耦的信息提取流程，包括SQUEEZE、RECOVER和RELABEL阶段。尽管这些方法可扩展到大规模数据集，但它们存在计算开销过高和跨架构泛化能力差的问题。在本文中，我们揭示了这些瓶颈的根本原因：从数据到模型再回到图像的隐式双压缩过程本质上会导致严重的信息损失。关键的是，我们通过经验和理论证明，这种损失会产生分布偏移，从根本上破坏了广泛采用的RELABEL策略，将预训练模型转变为产生次优标签的不可靠标注器。为了克服这些关键缺陷，我们提出了CIM，一种新颖的、度量驱动的框架，它摒弃了有缺陷的双压缩范式。相反，CIM明确量化并最小化原始数据集与合成数据集之间的信息差距。通过直接对齐数据分布，我们的方法确保了高保真信息压缩，并固有地满足了有效重标注的前提条件。大量实验表明，CIM建立了新的最先进水平。值得注意的是，它在单个RTX-4090 GPU上仅用80分钟就将ImageNet-1K蒸馏到IPC=10，在ResNet-18上达到了前所未有的48.7% Top-1准确率，分别比之前的SOTA方法NRR-DD和DELT显著高出2.6%和2.9%。我们的代码可在该https URL获取。
英文摘要：
Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
