超越特征相似性：基于类别感知条件互信息的高效数据集蒸馏
Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information
AI总结：
针对现有数据集蒸馏方法因依赖特征相似性度量导致合成数据难学习的问题，提出基于最小化类别感知条件互信息的新方法，可作为通用正则化手段提升现有方法的性能与训练效率。
摘要：
数据集蒸馏（DD）旨在通过创建一个规模更小、性能与完整真实数据集相当的合成数据集，降低在大型数据集上训练深度神经网络所需的时间与内存消耗。然而，当前的数据集蒸馏方法生成的合成数据集往往让网络难以学习，原因是这类方法通过特征相似性度量指标（例如分布匹配DM）压缩了原始数据中的大量信息。本研究引入条件互信息（CMI）来评估数据集的类别感知复杂度，并提出了一种通过最小化CMI实现的新方法。具体而言，我们在最小化蒸馏损失的同时，通过最小化合成数据集在预训练网络特征空间中的经验CMI，约束其类别感知复杂度。通过开展全面的实验，我们证明该方法可作为现有DD方法的通用正则化手段，能够提升性能与训练效率。
英文摘要：
Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks on large datasets, by creating a smaller synthetic dataset that has similar performance to that of the full real dataset. However, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metrics measuring feature similarity, e,g., distribution matching (DM). In this work, we introduce conditional mutual information (CMI) to assess the class-aware complexity of a dataset and propose a novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the class-aware complexity of the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as a general regularization method to existing DD methods and improve the performance and training efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
