arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.08332cs.CV
收藏

生成式去噪蒸馏:简单随机噪声诱导面向密集预测的高效知识迁移

Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction

Huazhong University of Science and Technology(华中科技大学);University of Michigan(密歇根大学)

Zhaoge Liu, Xiaohao Xu, Yunkang Cao, Weiming Shen

AI总结:

本文提出生成式去噪蒸馏GDD,通过向学生概念特征注入随机噪声并生成实例特征与教师对齐,在检测与分割任务上取得新SOTA。

摘要:

知识蒸馏是将知识从更强大的大模型(教师)迁移到更简单对应模型(学生)的过程。当前大量方法让学生直接模仿教师的知识。然而,这些主流方法学到的表示中仍存在冗余,它们倾向于不加区分地学习每个空间位置的特征。为了从教师中获得更紧凑的表示(概念特征),受人类认知启发,我们提出了一种创新方法,称为生成式去噪蒸馏(GDD),其中将随机噪声添加到学生的概念特征中,以将其嵌入到由浅层网络生成的实例特征中。然后,生成的实例特征与来自教师的实例知识对齐。我们在目标检测、实例分割和语义分割上进行了广泛实验,以证明我们方法的通用性和有效性。值得注意的是,GDD在上述任务中取得了新的最先进性能。通过增强基于ResNet-18的PspNet和DeepLabV3,我们在语义分割上取得了显著提升,mIoU分数分别达到74.67和77.69,超过了它们在Cityscapes数据集20个类别上此前69.85和73.20的分数。源代码可在https://github.com/ZhgLiu/GDD获取。

英文摘要:

Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly. However, redundancy still exists in the learned representations through these prevalent methods, which tend to learn each spatial location's features indiscriminately. To derive a more compact representation (concept feature) from the teacher, inspired by human cognition, we suggest an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The source code is available at https://github.com/ZhgLiu/GDD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑