AdvFunMatch：当一致性教学遇上对抗鲁棒性
AdvFunMatch: When Consistent Teaching Meets Adversarial Robustness
AI总结：
本文提出 AdvFunMatch 方法，通过在一致性教学框架下对 $\ell_p$-范数球内最坏情况样本进行函数匹配，实现对抗鲁棒性从教师模型到学生模型的有效迁移，同时保持高干净准确率。
摘要：
一致性教学是实现知识蒸馏（KD）的一种有效范式，其中学生模型和教师模型接收相同的输入，知识蒸馏被视作函数匹配任务（FunMatch）。然而，FunMatch 的一个局限在于它没有考虑对抗鲁棒性的迁移，即模型抵抗对抗攻击的能力。为了解决这一问题，我们提出了一种简单但有效的策略，称为对抗函数匹配（AdvFunMatch），其目标是在一致性教学的框架下，对训练数据的 $\ell_p$-范数球内所有数据点进行分布匹配。AdvFunMatch 被形式化为一个最小-最大优化问题，它识别出使教师模型与学生模型输出之间 KL 散度最大化的最坏情况实例，我们称之为“不匹配样本”，然后在这些不匹配样本上匹配输出。我们的实验结果表明，AdvFunMatch 能够有效地训练出同时具有高干净准确率和鲁棒性的学生模型。此外，我们揭示了强数据增强（例如 AutoAugment）在 AdvFunMatch 中是有益的，而先前的工作发现它们在对抗训练中效果较差。代码可在 \url{https://gitee.com/zihui998/adv-fun-match} 获取。
英文摘要：
\emph{Consistent teaching} is an effective paradigm for implementing knowledge distillation (KD), where both student and teacher models receive identical inputs, and KD is treated as a function matching task (FunMatch). However, one limitation of FunMatch is that it does not account for the transfer of adversarial robustness, a model's resistance to adversarial attacks. To tackle this problem, we propose a simple but effective strategy called Adversarial Function Matching (AdvFunMatch), which aims to match distributions for all data points within the $\ell_p$-norm ball of the training data, in accordance with consistent teaching. Formulated as a min-max optimization problem, AdvFunMatch identifies the worst-case instances that maximizes the KL-divergence between teacher and student model outputs, which we refer to as "mismatched examples," and then matches the outputs on these mismatched examples. Our experimental results show that AdvFunMatch effectively produces student models with both high clean accuracy and robustness. Furthermore, we reveal that strong data augmentations (\emph{e.g.}, AutoAugment) are beneficial in AdvFunMatch, whereas prior works have found them less effective in adversarial training. Code is available at \url{https://gitee.com/zihui998/adv-fun-match}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
