arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.07014cs.LG
收藏

数据集蒸馏:全面综述

Dataset Distillation: A Comprehensive Review

National University of Singapore(新加坡国立大学)

Ruonan Yu, Songhua Liu, Xinchao Wang

AI总结:

针对深度学习中海量数据带来的存储传输、训练效率及隐私版权问题,本文对数据集蒸馏(DD)技术的最新进展展开全面综述,提出统一算法框架、系统分类方法,分析理论关联,指出当前挑战并展望未来方向。

摘要:

深度学习近期的成功很大程度上归功于用于训练深度神经网络的海量数据。尽管取得了前所未有的成功,但遗憾的是,海量数据显著增加了存储和传输负担,还进一步导致模型训练过程繁琐。此外,依赖原始数据进行训练本身也引发了隐私和版权方面的担忧。为缓解这些缺陷,数据集蒸馏(DD,也称为数据集浓缩(DC))被提出,且近期在学界吸引了大量研究关注。给定一个原始数据集,DD的目标是生成一个包含合成样本的小得多的数据集,基于该小数据集训练的模型能取得与在原始数据集上训练的模型相当的性能。本文对DD及其应用的最新进展进行了全面综述与总结。我们首先正式介绍了该任务,并提出了所有现有DD方法遵循的总体算法框架。接着,我们对该领域当前的方法进行了系统分类,并讨论了它们之间的理论关联。我们还通过大量实验指出了DD当前面临的挑战,并展望了未来工作的可能方向。

英文摘要:

Recent success of deep learning is largely attributed to the sheer amount of data used for training deep neural networks.Despite the unprecedented success, the massive data, unfortunately, significantly increases the burden on storage and transmission and further gives rise to a cumbersome model training process. Besides, relying on the raw data for training \emph{per se} yields concerns about privacy and copyright. To alleviate these shortcomings, dataset distillation~(DD), also known as dataset condensation (DC), was introduced and has recently attracted much research attention in the community. Given an original dataset, DD aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. In this paper, we give a comprehensive review and summary of recent advances in DD and its application. We first introduce the task formally and propose an overall algorithmic framework followed by all existing DD methods. Next, we provide a systematic taxonomy of current methodologies in this area, and discuss their theoretical interconnections. We also present current challenges in DD through extensive experiments and envision possible directions for future works.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑