函数一致的特征蒸馏
Function-Consistent Feature Distillation
AI总结：
针对L2特征距离忽视网络各向异性处理的问题，本文提出FCFD，通过约束师生特征经后续网络产生相似输出来实现函数一致蒸馏，并在分类和检测任务上验证其性能与可组合增益。
摘要：
特征蒸馏使学生模仿教师的中间特征。几乎所有现有特征蒸馏方法都使用L2距离或其轻微变体作为教师与学生特征之间的距离度量。然而，尽管L2距离对所有维度都是各向同性的，神经网络在不同维度上的操作通常是各向异性的，即具有相同2范数但位于中间特征不同维度的扰动会导致最终输出发生幅度差异很大的变化。鉴于此，我们认为教师与学生特征之间的相似性不应仅基于其外观（即L2距离）来衡量，更重要的是应通过它们在函数上的差异来衡量，也就是网络后续层将如何读取、解码并处理它们。因此，我们提出了函数一致的特征蒸馏（Function-Consistent Feature Distillation，FCFD），它显式优化教师与学生特征之间的函数相似性。FCFD的核心思想是使教师和学生特征不仅在数值上相似，更重要的是在输入同一网络的后续部分时产生相似输出。借助FCFD，学生能更忠实地模仿教师，并从教师处学到更多。在图像分类和目标检测上的大量实验证明了FCFD相对于现有方法的优越性。此外，我们可以将FCFD与许多现有方法结合，以获得更高的准确率。我们的代码可在https://github.com/LiuDongyang6/FCFD获取。
英文摘要：
Feature distillation makes the student mimic the intermediate features of the teacher. Nearly all existing feature-distillation methods use L2 distance or its slight variants as the distance metric between teacher and student features. However, while L2 distance is isotropic w.r.t. all dimensions, the neural network's operation on different dimensions is usually anisotropic, i.e., perturbations with the same 2-norm but in different dimensions of intermediate features lead to changes in the final output with largely different magnitude. Considering this, we argue that the similarity between teacher and student features should not be measured merely based on their appearance (i.e., L2 distance), but should, more importantly, be measured by their difference in function, namely how later layers of the network will read, decode, and process them. Therefore, we propose Function-Consistent Feature Distillation (FCFD), which explicitly optimizes the functional similarity between teacher and student features. The core idea of FCFD is to make teacher and student features not only numerically similar, but more importantly produce similar outputs when fed to the later part of the same network. With FCFD, the student mimics the teacher more faithfully and learns more from the teacher. Extensive experiments on image classification and object detection demonstrate the superiority of FCFD to existing methods. Furthermore, we can combine FCFD with many existing methods to obtain even higher accuracy. Our codes are available at https://github.com/LiuDongyang6/FCFD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
