arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.03295cs.CV
收藏

预训练模型能否辅助数据集蒸馏?

Can pre-trained models assist in dataset distillation?

Zhejiang University of Technology(浙江工业大学);University of Electronic Science and Technology of China(电子科技大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学)

Yao Lu, Xuguang Chen, Yuchen Zhang, Jianyang Gu, Tianle Zhang, Yifan Zhang, Xiaoniu Yang, Qi Xuan, Kai Wang, Yang You

AI总结:

该研究探讨预训练模型对数据集蒸馏的辅助作用,系统分析预训练模型的初始化参数、架构、训练轮次等因素的影响,通过最优选择显著提升跨架构泛化能力,为数据集蒸馏技术发展提供参考。

摘要:

数据集蒸馏(Dataset Distillation,DD)是一种将大规模原始数据集中的知识封装到小型合成数据集中以实现高效训练的重要技术。同时,预训练模型(Pre-trained Models,PTMs)充当知识存储库,包含来自原始数据集的大量信息。这自然引出一个问题:PTMs能否有效地将知识迁移到合成数据集,准确指导DD?为此,我们开展了初步实验,证实了PTMs对DD的贡献。随后,我们系统研究了PTMs中的不同选择,包括初始化参数、模型架构、训练轮次和领域知识,发现:1)增加模型多样性可提升合成数据集的性能;2)次优模型也能辅助DD,且在某些情况下优于训练充分的模型;3)特定领域的PTMs对DD并非必需,但合理的领域匹配至关重要。最后,通过选择最优选项,我们显著提升了跨架构泛化能力,优于基线DD方法。我们希望这项工作能助力研究人员开发更优的DD技术。代码可在https://github.com/yaolu-zjut/DDInterpreter获取。

英文摘要:

Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑