arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1911.03080cs.LGstat.ML
收藏

基于图的深度几何知识蒸馏

Deep geometric knowledge distillation with graphs

IMT Atlantique(大西洋国立高等矿业电信学校);University of Southern California(南加州大学);Mila(米拉研究所);HEC Montréal(蒙特利尔高等商学院)

Carlos Lassance, Myriam Bontonou, Ghouthi Boukli Hacene, Vincent Gripon, Jian Tang, Antonio Ortega

摘要:

在大多数情况下,深度学习架构的训练与操作量和能耗无关。然而,一些应用程序,如嵌入式系统,在推理过程中可能会受到资源约束。减少深度学习架构规模的一种流行方法是将知识从较大的网络(教师)提取到较小的网络(学生)。直接训练学生模仿教师的表征可能是有效的,但它需要两者共享相同的潜在空间维度。在这项工作中,我们专注于相对知识蒸馏(RKD),它考虑了各自的潜在空间的几何形状,允许知识的维度不可知转移。具体来说,我们介绍了一个基于图的RKD方法,其中图是用来捕捉潜在空间的几何形状。使用经典的计算机视觉基准,我们证明了所提出的方法,有效地蒸馏知识从教师到学生的能力,导致更好的准确性为相同的预算相比,现有的RKD替代品。

英文摘要:

In most cases deep learning architectures are trained disregarding the amount of operations and energy consumption. However, some applications, like embedded systems, can be resource-constrained during inference. A popular approach to reduce the size of a deep learning architecture consists in distilling knowledge from a bigger network (teacher) to a smaller one (student). Directly training the student to mimic the teacher representation can be effective, but it requires that both share the same latent space dimensions. In this work, we focus instead on relative knowledge distillation (RKD), which considers the geometry of the respective latent spaces, allowing for dimension-agnostic transfer of knowledge. Specifically we introduce a graph-based RKD method, in which graphs are used to capture the geometry of latent spaces. Using classical computer vision benchmarks, we demonstrate the ability of the proposed method to efficiently distillate knowledge from the teacher to the student, leading to better accuracy for the same budget as compared to existing RKD alternatives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑