arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.05475cs.LGcs.MM
收藏

基于群体计数的评审机制缩小知识蒸馏中的能力差距

Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd Counting

Jiangxi Normal University(江西师范大学);East China University of Science and Technology(华东理工大学)

Yunxin Liu, Qiaosi Yi, Jinshan Zeng

摘要:

轻量级人群计数模型,特别是基于知识蒸馏(KD)的模型,由于其在计算效率和硬件需求上的优势,近年来受到了越来越多的关注。然而,现有的基于KD的模型通常遭受容量差距问题,导致学生网络的性能受到教师网络的限制。在本文中,我们通过引入一种新的审查机制KD模型,在研究过程中的人类审查机制的动机,解决这个问题。因此,所提出的模型被称为ReviewKD。该模型包括一个教学阶段和一个回顾阶段,我们首先利用一个训练有素的重型教师网络的潜在特征转移到一个轻量级的学生网络在教学阶段,然后在回顾阶段产生一个精细的估计密度图的基础上学习的功能,通过审查机制。通过与最先进的模型进行比较,在六个基准数据集上的一组实验证明了ReviewKD的有效性。数值实验结果表明,ReviewKD在人群计数方面优于现有的轻量级模型,能够有效缓解容量缺口问题,尤其是在教师网络方面具有超越教师网络的性能。除了轻量级模型,我们还表明,建议的审查机制可以作为一个即插即用模块,以进一步提高性能的一种沉重的人群计数模型,而无需修改神经网络架构和引入任何额外的模型参数。

英文摘要:

The lightweight crowd counting models, in particular knowledge distillation (KD) based models, have attracted rising attention in recent years due to their superiority on computational efficiency and hardware requirement. However, existing KD based models usually suffer from the capacity gap issue, resulting in the performance of the student network being limited by the teacher network. In this paper, we address this issue by introducing a novel review mechanism following KD models, motivated by the review mechanism of human-beings during the study. Thus, the proposed model is dubbed ReviewKD. The proposed model consists of an instruction phase and a review phase, where we firstly exploit a well-trained heavy teacher network to transfer its latent feature to a lightweight student network in the instruction phase, then in the review phase yield a refined estimate of the density map based on the learned feature through a review mechanism. The effectiveness of ReviewKD is demonstrated by a set of experiments over six benchmark datasets via comparing to the state-of-the-art models. Numerical results show that ReviewKD outperforms existing lightweight models for crowd counting, and can effectively alleviate the capacity gap issue, and particularly has the performance beyond the teacher network. Besides the lightweight models, we also show that the suggested review mechanism can be used as a plug-and-play module to further boost the performance of a kind of heavy crowd counting models without modifying the neural network architecture and introducing any additional model parameter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑