arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.09670cs.LGcs.CRcs.CVstat.ML
收藏

RobustBench:标准化对抗鲁棒性基准

RobustBench: a standardized adversarial robustness benchmark

Univ. of Tübingen(图宾根大学);EPFL(洛桑联邦理工学院);Princeton Univ(普林斯顿大学);Purdue Univ.(普渡大学)

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein

摘要:

作为一个研究社区,我们仍然缺乏对对抗鲁棒性进展的系统性理解,这通常使得很难确定训练鲁棒模型的最有前途的想法。鲁棒性基准测试中的一个关键挑战是其评估往往容易出错,导致鲁棒性高估。我们的目标是建立一个对抗鲁棒性的标准化基准,在合理的计算预算内尽可能准确地反映所考虑模型的鲁棒性。为此,我们从考虑图像分类任务开始,并对允许的模型引入限制(将来可能会放宽)。我们使用AutoAttack评估对抗鲁棒性,AutoAttack是一种白盒和黑盒攻击的集合,最近在一项大规模研究中显示,与原始出版物相比,几乎所有的鲁棒性评估都有所改善。为了防止新防御过度适应AutoAttack,我们欢迎基于自适应攻击的外部评估,特别是在AutoAttack标记了潜在的高估鲁棒性的情况下。我们的排行榜位于https://robustbench.github.io/,包含对120多个模型的评估,旨在反映当前图像分类的最新技术水平,这些图像分类涉及$\ell_\infty$和$\ell_2$威胁模型中定义良好的一组任务以及常见的损坏,并在未来可能进行扩展。此外,我们还开源了库https://github.com/RobustBench/robustbench,该库提供对80多个强大模型的统一访问,以促进其下游应用程序。最后,基于收集到的模型,分析了鲁棒性对分布偏移、校准、分布外检测、公平性、隐私泄露、平滑性和可转移性等性能的影响。

英文摘要:

As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models. A key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget. To this end, we start by considering the image classification task and introduce restrictions (possibly loosened in the future) on the allowed models. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks, which was recently shown in a large-scale study to improve almost all robustness evaluations compared to the original publications. To prevent overadaptation of new defenses to AutoAttack, we welcome external evaluations based on adaptive attacks, especially where AutoAttack flags a potential overestimation of robustness. Our leaderboard, hosted at https://robustbench.github.io/, contains evaluations of 120+ models and aims at reflecting the current state of the art in image classification on a set of well-defined tasks in $\ell_\infty$- and $\ell_2$-threat models and on common corruptions, with possible extensions in the future. Additionally, we open-source the library https://github.com/RobustBench/robustbench that provides unified access to 80+ robust models to facilitate their downstream applications. Finally, based on the collected models, we analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑