arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.01601cs.CVcs.CR
收藏

RobustBench/AutoAttack 是否适合作为对抗鲁棒性的基准?

Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?

Fraunhofer Research Center Machine Learning(弗劳恩霍夫机器学习研究中心);Heidelberg University(海德堡大学);Fraunhofer ITWM(弗劳恩霍夫工业数学研究所);University of Siegen(锡根大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Saarland Informatics Campus(萨尔兰信息学园区);Institute for Machine Learning and Analytics (IMLA)(机器学习与分析研究所);Offenburg University(奥芬堡大学)

Peter Lorenz, Dominik Strassel, Margret Keuper, Janis Keuper

AI总结:

本文通过大量实验质疑 RobustBench/AutoAttack 作为对抗鲁棒性通用基准的适用性,指出其 l-inf、eps=8/255 扰动过强且易被检测,并发现低分辨率结果难以推广到高分辨率图像。

摘要:

近年来,RobustBench(Croce et al. 2020)已成为图像分类网络对抗鲁棒性领域被广泛认可的基准。在其最常被报告的子任务中,RobustBench 在 CIFAR10 上评估并排名已训练神经网络的对抗鲁棒性,采用 AutoAttack(Croce and Hein 2020b),施加 l-inf 扰动并限制 eps = 8/255。由于当前表现最好的模型领先分数约为基线的 60%,可以公平地将该基准描述为相当具有挑战性。尽管它在近期文献中被普遍接受,我们仍旨在促进关于 RobustBench 作为鲁棒性关键指标是否适合推广到实际应用的讨论。我们反对这一做法的论证有两方面,并由本文中呈现的大量实验支持:我们认为 I) AutoAttack 以 l-inf、eps = 8/255 对数据的改动强得不切实际,导致即使是简单的检测算法和人类观察者也能获得接近完美的对抗样本检测率。我们还表明,其他攻击方法在取得相似成功率的同时更难被检测。II) 在 CIFAR10 等低分辨率数据集上的结果不能很好地推广到更高分辨率的图像,因为基于梯度的攻击似乎随着分辨率提高而变得更加可检测。

英文摘要:

Recently, RobustBench (Croce et al. 2020) has become a widely recognized benchmark for the adversarial robustness of image classification networks. In its most commonly reported sub-task, RobustBench evaluates and ranks the adversarial robustness of trained neural networks on CIFAR10 under AutoAttack (Croce and Hein 2020b) with l-inf perturbations limited to eps = 8/255. With leading scores of the currently best performing models of around 60% of the baseline, it is fair to characterize this benchmark to be quite challenging. Despite its general acceptance in recent literature, we aim to foster discussion about the suitability of RobustBench as a key indicator for robustness which could be generalized to practical applications. Our line of argumentation against this is two-fold and supported by excessive experiments presented in this paper: We argue that I) the alternation of data by AutoAttack with l-inf, eps = 8/255 is unrealistically strong, resulting in close to perfect detection rates of adversarial samples even by simple detection algorithms and human observers. We also show that other attack methods are much harder to detect while achieving similar success rates. II) That results on low-resolution data sets like CIFAR10 do not generalize well to higher resolution images as gradient-based attacks appear to become even more detectable with increasing resolutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑