自监督数据集蒸馏：良好的压缩是你所需的全部
Self-supervised Dataset Distillation: A Good Compression Is All You Need
AI总结：
针对数据集蒸馏任务，提出基于自监督压缩的SC-DD框架，利用自监督模型BN统计量的大方差提升合成信息性，在多个数据集上以大模型实现了优于现有监督蒸馏方法的性能。
摘要：
数据集蒸馏旨在将大规模原始数据集中的信息压缩到一个新的紧凑数据集中，同时尽可能保留原始数据的最大信息本质。以往的研究主要集中在对齐原始数据与蒸馏数据之间的中间统计量，例如权重轨迹、特征、梯度、批归一化（BatchNorm）等。在本研究中，我们从原始数据集预训练阶段的模型信息性这一新视角出发来解决该任务。我们观察到，对于此前的最优方法SRe$^2$L，随着模型尺寸增大，有监督预训练模型在数据合成过程中恢复已学习信息的难度越来越高，因为模型内部的通道均值和方差趋于平坦，信息含量更低。我们进一步注意到，自监督模型的批归一化（BN）统计量具有更大的方差，能够提供更大的损失信号，通过梯度更新恢复的数据，在合成过程中具备更高的信息性。基于这一观察，我们提出了SC-DD，一种简单而有效的自监督压缩数据集蒸馏（Self-supervised Compression framework for Dataset Distillation）框架，与传统的有监督学习方案相比，它能促进多样化的信息压缩与恢复，还能进一步挖掘大型预训练模型的潜力，提升其性能。我们在CIFAR-100、Tiny-ImageNet和ImageNet-1K数据集上开展了大量实验，以验证所提方法的优越性。在相同的恢复和后训练预算下，当采用更大的模型时，所提出的SC-DD大幅优于所有此前的最优有监督数据集蒸馏方法，例如SRe$^2$L、MTT、TESLA、DC、CAFE等。代码可在https://github.com/VILA-Lab/SRe2L/tree/main/SCDD/ 获取。
英文摘要：
Dataset distillation aims to compress information from a large-scale original dataset to a new compact dataset while striving to preserve the utmost degree of the original data informational essence. Previous studies have predominantly concentrated on aligning the intermediate statistics between the original and distilled data, such as weight trajectory, features, gradient, BatchNorm, etc. In this work, we consider addressing this task through the new lens of model informativeness in the compression stage on the original dataset pretraining. We observe that with the prior state-of-the-art SRe$^2$L, as model sizes increase, it becomes increasingly challenging for supervised pretrained models to recover learned information during data synthesis, as the channel-wise mean and variance inside the model are flatting and less informative. We further notice that larger variances in BN statistics from self-supervised models enable larger loss signals to update the recovered data by gradients, enjoying more informativeness during synthesis. Building on this observation, we introduce SC-DD, a simple yet effective Self-supervised Compression framework for Dataset Distillation that facilitates diverse information compression and recovery compared to traditional supervised learning schemes, further reaps the potential of large pretrained models with enhanced capabilities. Extensive experiments are conducted on CIFAR-100, Tiny-ImageNet and ImageNet-1K datasets to demonstrate the superiority of our proposed approach. The proposed SC-DD outperforms all previous state-of-the-art supervised dataset distillation methods when employing larger models, such as SRe$^2$L, MTT, TESLA, DC, CAFE, etc., by large margins under the same recovery and post-training budgets. Code is available at https://github.com/VILA-Lab/SRe2L/tree/main/SCDD/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
