通过均衡多教师对抗蒸馏缓解准确率-鲁棒性权衡
Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation
AI总结：
本文提出均衡多教师对抗鲁棒性蒸馏（B-MTARD），通过强干净教师与强鲁棒教师分别处理干净和对抗样本，并设计熵均衡与归一化损失均衡算法，有效缓解准确率与鲁棒性之间的权衡，在多个数据集上超越现有方法。
摘要：
对抗训练是提高深度神经网络抵御对抗攻击鲁棒性的一种实用方法。尽管对抗训练能带来可靠的鲁棒性，但其对干净样本的性能会受到负面影响，这意味着准确率与鲁棒性之间存在权衡。近年来，一些研究尝试在对抗训练中使用知识蒸馏方法，在提升鲁棒性方面取得了有竞争力的表现，但对干净样本的准确率仍然有限。在本文中，为了缓解准确率-鲁棒性权衡，我们提出了均衡多教师对抗鲁棒性蒸馏（B-MTARD），通过引入一个强干净教师和一个强鲁棒教师分别处理干净样本和对抗样本，来指导模型的对抗训练过程。在优化过程中，为了确保不同教师具有相似的知识尺度，我们设计了基于熵的均衡算法来调整教师的温度，并保持教师信息熵一致。此外，为了确保学生从多个教师处获得相对一致的学习速度，我们提出了归一化损失均衡算法来调整不同类型知识的学习权重。在三个公开数据集上进行的一系列实验表明，B-MTARD在面对各种对抗攻击时优于当前最先进的方法。
英文摘要：
Adversarial Training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. Although bringing reliable robustness, the performance towards clean examples is negatively affected after Adversarial Training, which means a trade-off exists between accuracy and robustness. Recently, some studies have tried to use knowledge distillation methods in Adversarial Training, achieving competitive performance in improving the robustness but the accuracy for clean samples is still limited. In this paper, to mitigate the accuracy-robustness trade-off, we introduce the Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD) to guide the model's Adversarial Training process by applying a strong clean teacher and a strong robust teacher to handle the clean examples and adversarial examples, respectively. During the optimization process, to ensure that different teachers show similar knowledge scales, we design the Entropy-Based Balance algorithm to adjust the teacher's temperature and keep the teachers' information entropy consistent. Besides, to ensure that the student has a relatively consistent learning speed from multiple teachers, we propose the Normalization Loss Balance algorithm to adjust the learning weights of different types of knowledge. A series of experiments conducted on three public datasets demonstrate that B-MTARD outperforms the state-of-the-art methods against various adversarial attacks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
