发表机构
Islamic Azad University; Iran University of Science and Technology; Meta（伊斯兰阿扎德大学; 伊朗科技大学; Meta）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
RAPID通过分离可靠性门控目标与自适应样本对提议，实现高效的关系蒸馏，在AG News和SST-2上取得最优平均准确率，验证了目标可靠性与评估优先级可分离的设计。
AI 中文摘要
样本间关系蒸馏通过匹配小批量内样本之间的关系来迁移教师模型的表征几何结构。计算所有样本对的时间复杂度与批量大小呈二次方关系，而均匀子采样可能无法高效利用有限的关系预算。我们提出了可靠性感知的样本对重要性蒸馏（Reliability Aware Pair Importance Distillation，简称RAPID），该方法将可靠性门控的关系目标与全支持的自适应样本对提议分离。可靠性决定了哪些教师关系被强调，而校准后的教师熵和分离的学生-教师残差决定了评估哪些关系。精确的逆提议校正使得损失和梯度估计器相对于门控的小批量目标具有条件无偏性。我们在两个文本分类设置中评估了RAPID：使用三对配对种子和256的关系预算在AG News上进行BERT到DistilBERT的蒸馏，以及使用三对配对种子和64的关系预算在SST-2上进行DistilBERT到DistilBERT的蒸馏。可靠性门控的关系蒸馏在两个数据集上均取得了最高的平均学生准确率：AG News上为94.285±0.054%，SST-2上为88.800±0.532%。RAPID排名第二，分别达到94.241±0.025%和88.685±0.462%，而交叉熵基线为94.154±0.124%和87.271±0.162%。试点评估计入与主要关系评估相同的总预算。在这两种设置中，门控目标产生了最高的平均准确率，而自适应提议保持在种子级变异范围内。这些结果支持了模块化的观点，即目标可靠性和评估优先级是可分离的设计维度。
英文摘要
Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair Importance Distillation, or RAPID, which separates a reliability gated relational target from a full support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated. Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT to DistilBERT distillation using three paired seeds and a relation budget of 64. Reliability gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 plus or minus 0.054 percent on AG News and 88.800 plus or minus 0.532 percent on SST-2. RAPID ranks second, achieving 94.241 plus or minus 0.025 percent and 88.685 plus or minus 0.462 percent, respectively, compared with 94.154 plus or minus 0.124 percent and 87.271 plus or minus 0.162 percent for the cross entropy baseline. Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.
