arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1905.00877stat.MLcs.LGmath.OC
收藏

你只需要学习一次:通过最大化原则加速对抗训练

You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

Peking University(北京大学);Stanford University(斯坦福大学);Beijing Institute of Big Data Research(北京大数据研究院)

Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, Bin Dong

摘要:

深度学习在计算机视觉和自然语言处理的许多任务中取得了最先进的成果。然而,最近的研究表明,深度网络可能容易受到对抗性扰动的影响,这引发了深度网络的严重鲁棒性问题。对抗性训练通常被表述为鲁棒优化问题,是提高深度网络鲁棒性的有效方法。现有对抗训练算法的一个主要缺点是生成对抗示例的计算开销,通常远大于网络训练的计算开销。这导致对抗训练的整体计算成本难以承受。在本文中,我们证明了对抗训练可以被视为离散时间微分博弈。通过分析该问题的Pontryagin最大值原理(PMP),我们观察到对手的更新只与网络第一层的参数相耦合。这启发我们在对手更新期间将大部分前向和反向传播限制在网络的第一层内。这有效地减少了完整的前向和后向传播的总数,只有一个对手更新的每一组。因此,我们将此算法称为YOPO(You Only Exclusive Once)。数值实验表明,该算法仅需投影梯度下降(PGD)算法1/5 ~ 1/4的GPU时间,就能达到相当的防御精度。我们的代码可在https://www.example.com上获得。https://github.com/a1600012888/YOPO-You-Only-Propagate-Once

英文摘要:

Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations, which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to the unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin's Maximal Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (You Only Propagate Once). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with approximately 1/5 ~ 1/4 GPU time of the projected gradient descent (PGD) algorithm. Our codes are available at https://https://github.com/a1600012888/YOPO-You-Only-Propagate-Once.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑