GIFT：以近乎零成本释放蒸馏数据集中标签的全部潜力
GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost
AI总结：
本文提出即插即用方法GIFT，通过软标签精炼和余弦相似度损失函数，在近乎零额外计算成本下充分利用标签信息，显著提升了多种规模数据集蒸馏方法的性能及跨优化器泛化能力。
摘要：
数据集蒸馏的最新进展表明，使用预训练教师模型生成的软标签具有显著优势。本文通过强调标签的充分利用，引入了一个新视角。我们首先对数据集蒸馏中用于软标签利用的各种损失函数进行了全面比较，发现在合成数据集上训练的模型对软标签利用的损失函数选择表现出高度敏感性。这一发现突显了在合成数据集上训练模型时使用通用损失函数的必要性。基于这些见解，我们引入了一种极其简单但极为有效的即插即用方法GIFT，该方法包含软标签精炼和基于余弦相似度的损失函数，以高效利用完整的标签信息。大量实验表明，GIFT在各种数据集规模下均能持续提升最先进的数据集蒸馏方法，且不产生额外的计算成本。重要的是，GIFT显著增强了跨优化器泛化能力，这是一个此前被忽视的领域。例如，在ImageNet-1K且IPC = 10的设置下，GIFT将最先进方法RDED的跨优化器泛化能力提升了30.8%。代码可在 https://github.com/LINs-lab/GIFT 获取。
英文摘要：
Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models. In this paper, we introduce a novel perspective by emphasizing the full utilization of labels. We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. This finding highlights the necessity of a universal loss function for training models on synthetic datasets. Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. Extensive experiments indicate that GIFT consistently enhances state-of-the-art dataset distillation methods across various dataset scales, without incurring additional computational costs. Importantly, GIFT significantly enhances cross-optimizer generalization, an area previously overlooked. For instance, on ImageNet-1K with IPC = 10, GIFT enhances the state-of-the-art method RDED by 30.8% in cross-optimizer generalization. Our code is available at https://github.com/LINs-lab/GIFT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
