arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.13850cs.LGcs.AI
收藏

基于模型反转的压缩数据扩展用于知识蒸馏

Condensed Data Expansion Using Model Inversion for Knowledge Distillation

Kuluhan Binici, Shivam Aggarwal, Cihan Acar, Nam Trung Pham, Karianto Leman, Gim Hee Lee, Tulika Mitra

AI总结:

本文提出利用模型反转生成合成数据来扩展压缩数据集,以改善知识蒸馏中学生模型的准确率,在多个数据集和架构上比标准方法提升最高11.4%,并在少样本场景下依然有效。

摘要:

压缩数据集提供了更大数据集的紧凑表示,但直接在其上训练模型或通过知识蒸馏(KD)利用它们提升模型性能,可能因信息有限而导致次优结果。为解决这一问题,我们提出一种方法,利用模型反转扩展压缩数据集,模型反转是一种基于预训练模型对其训练数据的印象生成合成数据的技术。该方法特别适用于知识蒸馏场景,因为教师模型已经预训练并保留了原始训练数据的知识。通过生成补充压缩样本的合成数据,我们丰富了训练集并更好地逼近底层数据分布,从而在知识蒸馏过程中提升学生模型的准确率。我们的方法相比仅使用压缩数据集在知识蒸馏准确率上取得了显著提升,并在多个数据集和模型架构上比标准的基于模型反转的知识蒸馏方法高出最多11.4%。重要的是,即使每个类别仅使用一个压缩样本,该方法仍然有效,并且在仅有少量真实数据样本可用的少样本场景中也能提升性能。

英文摘要:

Condensed datasets offer a compact representation of larger datasets, but training models directly on them or using them to enhance model performance through knowledge distillation (KD) can result in suboptimal outcomes due to limited information. To address this, we propose a method that expands condensed datasets using model inversion, a technique for generating synthetic data based on the impressions of a pre-trained model on its training data. This approach is particularly well-suited for KD scenarios, as the teacher model is already pre-trained and retains knowledge of the original training data. By creating synthetic data that complements the condensed samples, we enrich the training set and better approximate the underlying data distribution, leading to improvements in student model accuracy during knowledge distillation. Our method demonstrates significant gains in KD accuracy compared to using condensed datasets alone and outperforms standard model inversion-based KD methods by up to 11.4% across various datasets and model architectures. Importantly, it remains effective even when using as few as one condensed sample per class, and can also enhance performance in few-shot scenarios where only limited real data samples are available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑