arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.04338cs.LG
收藏

回归神经网络中无数据知识蒸馏的合成数据生成方法

Synthetic data generation method for data-free knowledge distillation in regression neural networks

Bioinformatics Institute, Singapore(新加坡生物信息学研究所)

Tianxun Zhou, Keng-Hwee Chiam

AI总结:

本研究提出一种新的合成数据生成策略,通过优化学生与教师模型间较大但有界的差异,在无原始数据条件下提升回归任务知识蒸馏中学生模型的性能。

摘要:

知识蒸馏是一种将较大的神经网络(称为教师模型)压缩为较小的神经网络(称为学生模型)的技术,同时尽可能保持较大神经网络的性能。现有的知识蒸馏方法大多适用于分类任务,且许多方法需要访问用于训练教师模型的数据。为了解决在原始训练数据缺失情况下回归任务的知识蒸馏问题,先前的工作提出了一种无数据知识蒸馏方法,其中使用对抗性训练于学生模型的生成器模型生成合成数据,这些合成数据及其由教师模型预测的标签随后用于训练学生模型。在本研究中,我们考察了各种合成数据生成方法的行为,并提出了一种新的合成数据生成策略,该策略直接优化学生模型与教师模型之间较大但有界的差异。我们在基准实验和案例研究实验上的结果表明,所提出的策略使学生模型能够更好地学习,并更紧密地模仿教师模型的性能。

英文摘要:

Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger neural network as much as possible. Existing methods of knowledge distillation are mostly applicable for classification tasks. Many of them also require access to the data used to train the teacher model. To address the problem of knowledge distillation for regression tasks under the absence of original training data, previous work has proposed a data-free knowledge distillation method where synthetic data are generated using a generator model trained adversarially against the student model. These synthetic data and their labels predicted by the teacher model are then used to train the student model. In this study, we investigate the behavior of various synthetic data generation methods and propose a new synthetic data generation strategy that directly optimizes for a large but bounded difference between the student and teacher model. Our results on benchmark and case study experiments demonstrate that the proposed strategy allows the student model to learn better and emulate the performance of the teacher model more closely.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑