用神经正切核重新思考对抗训练
Rethinking Adversarial Training with Neural Tangent Kernel
AI总结：
本文基于神经正切核分析对抗训练动力学，揭示数据归一化与无偏批归一化估计量的影响，提出更省时的对抗训练方法，并利用核频谱特征缓解灾难性过拟合。
摘要：
对抗训练（AT）是深度学习安全中一个重要且引人关注的主题，表现出许多神秘而奇异的性质。近年来基于神经正切核（Neural Tangent Kernel, NTK）的神经网络训练动力学研究，使得重新认识AT并深入分析其性质成为可能。本文利用NTK对AT过程及其性质进行了深入考察，例如NTK的演化。我们发现了三项此前工作遗漏的新发现。第一，我们揭示了数据归一化对AT的影响，以及批归一化层中无偏估计量的重要性。第二，我们通过实验探索了核动力学，并提出了更节省时间的AT方法。第三，我们研究了核内部的频谱特征，以解决灾难性过拟合问题。据我们所知，这是首个利用核动力学观察结果来改进现有AT方法的工作。
英文摘要：
Adversarial training (AT) is an important and attractive topic in deep learning security, exhibiting mysteries and odd properties. Recent studies of neural network training dynamics based on Neural Tangent Kernel (NTK) make it possible to reacquaint AT and deeply analyze its properties. In this paper, we perform an in-depth investigation of AT process and properties with NTK, such as NTK evolution. We uncover three new findings that are missed in previous works. First, we disclose the impact of data normalization on AT and the importance of unbiased estimators in batch normalization layers. Second, we experimentally explore the kernel dynamics and propose more time-saving AT methods. Third, we study the spectrum feature inside the kernel to address the catastrophic overfitting problem. To the best of our knowledge, it is the first work leveraging the observations of kernel dynamics to improve existing AT methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
