在紧凑空间中对齐：异构架构间的对比知识蒸馏
Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures
AI总结：
针对异构架构间特征蒸馏性能退化问题，提出基于低频分量提取与样本级对比学习的LFCC框架，在紧凑空间对齐特征，显著提升ImageNet-1K和CIFAR-100上的性能。
摘要：
知识蒸馏通常用于压缩神经网络，以降低推理成本和内存占用。在同构架构的场景下，基于特征的方法已被广泛验证其有效性。然而，在教师模型和学生模型为异构架构的场景中，特征表示的内在差异显著降低了这些方法的性能。近期研究强调，低频分量构成了图像特征的主体。受此启发，我们提出了一种基于低频分量的对比知识蒸馏（LFCC）框架，该框架显著提升了异构架构间基于特征的蒸馏性能。具体而言，我们设计了一组多尺度低通滤波器，用于提取教师和学生模型中间特征的低频分量，并在一个紧凑空间中对齐它们，以克服架构差异。此外，利用教师-学生框架固有的配对特性，我们设计了一种创新的样本级对比学习框架，巧妙地将样本内特征相似性和样本间特征差异性的约束重构为对比学习任务。这一策略使学生模型能够利用样本内特征一致性，同时增强不同样本间特征的判别性。因此，我们的LFCC框架准确捕捉了异构架构间特征表示的共性。在三种架构（CNN、Transformer和MLP）上的广泛评估和实证分析表明，LFCC在具有挑战性的ImageNet-1K和CIFAR-100基准上取得了优越的性能。所有代码将公开提供。
英文摘要：
Knowledge distillation is commonly employed to compress neural networks, reducing the inference costs and memory footprint. In the scenario of homogenous architecture, feature-based methods have been widely validated for their effectiveness. However, in scenarios where the teacher and student models are of heterogeneous architectures, the inherent differences in feature representation significantly degrade the performance of these methods. Recent studies have highlighted that low-frequency components constitute the majority of image features. Motivated by this, we propose a Low-Frequency Components-based Contrastive Knowledge Distillation (LFCC) framework that significantly enhances the performance of feature-based distillation between heterogeneous architectures. Specifically, we designe a set of multi-scale low-pass filters to extract the low-frequency components of intermediate features from both the teacher and student models, aligning them in a compact space to overcome architectural disparities. Moreover, leveraging the intrinsic pairing characteristic of the teacher-student framework, we design an innovative sample-level contrastive learning framework that adeptly restructures the constraints of within-sample feature similarity and between-sample feature divergence into a contrastive learning task. This strategy enables the student model to capitalize on intra-sample feature congruence while simultaneously enhancing the discrimination of features among disparate samples. Consequently, our LFCC framework accurately captures the commonalities in feature representation across heterogeneous architectures. Extensive evaluations and empirical analyses across three architectures (CNNs, Transformers, and MLPs) demonstrate that LFCC achieves superior performance on the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
