AI 中文总结
本文针对异构知识蒸馏中冗余抑制的缺陷，提出CoCaRS方法，通过CEE、SAC和ACR校准冗余抑制，在CIFAR-100等数据集上验证了其性能优势。
AI 中文摘要
知识蒸馏（KD）可让紧凑的学生模型从强大的教师模型中学习，已成为模型压缩的有效范式。多样模型架构的出现使KD从同构场景扩展到异构场景，但教师与学生模型间架构归纳偏置的差异常导致显著的表示偏差，限制了直接知识迁移的有效性。近期，冗余抑制通过保留跨架构不变性、对师生特征相关关系去相关以减少特征冗余，为异构KD提供了新视角。不过，该方法可能因均匀去相关削弱有用的结构信息，且固定系数会使冗余抑制的有效贡献对师生配对及训练阶段敏感。为解决这些问题，本文提出基于相关校准的冗余抑制方法CoCaRS，旨在更好地保留结构信息同时抑制冗余，并降低对系数设置的敏感性。具体而言，CoCaRS通过混淆证据估计（CEE）和强度分配控制（SAC）校准特征去相关：CEE用于捕获可靠语义关系以进行相关估计，SAC用于在去相关过程中保留判别结构；自适应系数调节（ACR）则根据校准后冗余抑制目标的相对损失规模调节其贡献，进一步降低对系数设置的敏感性。在CIFAR-100和ImageNet-1K上开展的大量实验验证了CoCaRS在提升蒸馏性能、降低对系数设置敏感性方面的有效性，代码将很快发布。
英文摘要
Knowledge distillation (KD) enables a compact student model to learn from a powerful teacher and has become an effective paradigm for model compression. The emergence of diverse model architectures has extended KD from homogeneous to heterogeneous settings. However, differences in architectural inductive biases between the teacher and student models often result in substantial representation discrepancies, limiting the effectiveness of direct knowledge transfer. Recently, redundancy suppression has offered a new perspective on heterogeneous KD by preserving cross-architecture invariance and reducing feature redundancy through decorrelation of teacher-student feature correlations. Nevertheless, this formulation may weaken useful structural information through uniform decorrelation, while a fixed coefficient may make the effective contribution of redundancy suppression sensitive to teacher-student pairs and training stages. To address these problems, Correlation Calibration-based Redundancy Suppression (CoCaRS) is proposed to better retain structural information while suppressing redundancy and reduce sensitivity to coefficient settings across teacher-student pairs and training stages. Specifically, CoCaRS calibrates feature decorrelation through Confusion Evidence Estimation (CEE) and Strength Allocation Control (SAC), which respectively capture reliable semantic relations for correlation estimation and preserve discriminative structure during decorrelation. Adaptive Coefficient Regulation (ACR) further regulates the contribution of the calibrated redundancy suppression objective according to its relative loss scale, reducing sensitivity to coefficient settings. Extensive experiments on CIFAR-100 and ImageNet-1K validate the effectiveness of CoCaRS in improving distillation performance and reducing sensitivity to coefficient settings. Code will be released soon.
