理解混合对抗训练中双批归一化的探索
Towards Understanding Dual BN In Hybrid Adversarial Training
AI总结：
本文研究混合对抗训练中双批归一化（Dual BN）的作用机制，发现解耦仿射参数比解耦统计量更关键，并提出双任务假设作为统一改进框架。
摘要：
在对抗训练（AT）中应用批归一化（BN）日益受到关注，尤其是当模型同时在对抗样本和干净样本上训练时（称为混合对抗训练，Hybrid-AT）。基于对抗样本和干净样本来自两个不同域的假设，先前工作中的常见做法是采用双批归一化（Dual BN），其中 BN 和 BN 分别用于对抗分支和干净分支。推动 Dual BN 的一个流行观点是，估计这种混合分布的归一化统计量具有挑战性，因此将其解耦以进行归一化可以获得更强的鲁棒性。与这一观点相反，我们揭示了在模型训练中，解耦统计量的作用不如解耦仿射参数重要。这一发现与先前工作（Rebuffi et al., 2023）一致，我们在其研究基础上进行了进一步探索。我们证明对抗样本和干净样本之间的域差距并不是很大，考虑到对抗扰动对模型准确率的显著影响，这一点是反直觉的。我们进一步提出了一个双任务假设，作为混合对抗训练改进的经验基础和统一框架。我们还研究了测试阶段的 Dual BN，并揭示仿射参数在推理过程中表征了鲁棒性。总体而言，我们的工作为理解混合对抗训练中 Dual BN 的机制及其潜在合理性提供了新的见解。
英文摘要：
There is a growing concern about applying batch normalization (BN) in adversarial training (AT), especially when the model is trained on both adversarial samples and clean samples (termed Hybrid-AT). With the assumption that adversarial and clean samples are from two different domains, a common practice in prior works is to adopt Dual BN, where BN and BN are used for adversarial and clean branches, respectively. A popular belief for motivating Dual BN is that estimating normalization statistics of this mixture distribution is challenging and thus disentangling it for normalization achieves stronger robustness. In contrast to this belief, we reveal that disentangling statistics plays a less role than disentangling affine parameters in model training. This finding aligns with prior work (Rebuffi et al., 2023), and we build upon their research for further investigations. We demonstrate that the domain gap between adversarial and clean samples is not very large, which is counter-intuitive considering the significant influence of adversarial perturbation on the model accuracy. We further propose a two-task hypothesis which serves as the empirical foundation and a unified framework for Hybrid-AT improvement. We also investigate Dual BN in test-time and reveal that affine parameters characterize the robustness during inference. Overall, our work sheds new light on understanding the mechanism of Dual BN in Hybrid-AT and its underlying justification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
