RobustBench: a standardized adversarial robustness benchmark
Univ. of Tübingen(图宾根大学);EPFL(洛桑联邦理工学院);Princeton Univ(普林斯顿大学);Purdue Univ.(普渡大学)
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, Matthias Hein
备注:
The camera-ready version accepted at the NeurIPS'21 Datasets and Benchmarks Track: 120+ evaluations, 80+ models, 7 leaderboards (Linf, L2, common corruptions; CIFAR-10, CIFAR-100, ImageNet), significantly expanded analysis part (calibration, fairness, privacy leakage, smoothness, transferability)
As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models. A key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget. To this end, we start by considering the image classification task and introduce restrictions (possibly loosened in the future) on the allowed models. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks, which was recently shown in a large-scale study to improve almost all robustness evaluations compared to the original publications. To prevent overadaptation of new defenses to AutoAttack, we welcome external evaluations based on adaptive attacks, especially where AutoAttack flags a potential overestimation of robustness. Our leaderboard, hosted at https://robustbench.github.io/, contains evaluations of 120+ models and aims at reflecting the current state of the art in image classification on a set of well-defined tasks in $\ell_\infty$- and $\ell_2$-threat models and on common corruptions, with possible extensions in the future. Additionally, we open-source the library https://github.com/RobustBench/robustbench that provides unified access to 80+ robust models to facilitate their downstream applications. Finally, based on the collected models, we analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?
Fraunhofer Research Center Machine Learning(弗劳恩霍夫机器学习研究中心);Heidelberg University(海德堡大学);Fraunhofer ITWM(弗劳恩霍夫工业数学研究所);University of Siegen(锡根大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Saarland Informatics Campus(萨尔兰信息学园区);Institute for Machine Learning and Analytics (IMLA)(机器学习与分析研究所);Offenburg University(奥芬堡大学)
Towards Deep Learning Models Resistant to Transfer-based Adversarial Attacks via Data-centric Robust Learning
School of Cyber Science and Engineering, Xi’an Jiaotong University(西安交通大学网络空间安全学院);School of Cyber Science and Technology, Zhejiang University(浙江大学网络空间安全学院)
Scaling Compute Is Not All You Need for Adversarial Robustness
ETH Zurich(苏黎世联邦理工学院);Georgia Institute of Technology(佐治亚理工学院);EPFL(洛桑联邦理工学院);Princeton University(普林斯顿大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)
Reliable Robustness Evaluation via Automatically Constructed Attack Ensembles
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology(南方科技大学可信自主系统研究院);Southern University of Science and Technology(南方科技大学)
On the Robustness of Adversarial Training Against Uncertainty Attacks
Department of Electric and Electronic Engineering, University of Cagliari(卡利亚里大学电子与电气工程系);Department of Informatics, Bioengineering, Robotics, and Systems Engineering, University of Genova(热那亚大学信息学、生物工程、机器人与系统工程系)
Semantically Guided Adversarial Testing of Vision Models Using Language Models
Institute of Theoretical and Applied Informatics, Polish Academy of Sciences(波兰科学院理论与应用信息学研究所);University of Lille, CNRS, Inria, Centrale Lille, UMR 9189 CRIStAL(里尔大学)
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Center for AI Safety(人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);Microsoft(微软)
Evaluating the Evaluators: Trust in Adversarial Robustness Tests
DIBRIS - Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa(热那亚大学信息学、生物工程、机器人与系统工程系);Department of Environmental Sciences, Informatics and Statistics, Ca' Foscari University of Venice(威尼斯卡弗里大学环境科学、信息学与统计学系)
An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection
Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development (GECAD), School of Engineering, Polytechnic of Porto (ISEP/IPP)(高级创新与发展智能工程与计算研究组(GECAD),波尔图理工学院工程学院(ISEP/IPP))
ShrimpXNet: A Transfer Learning Framework for Shrimp Disease Classification with Augmented Regularization, Adversarial Training, and Explainable AI
East West University(东西大学);Daffodil International University(达福迪尔国际大学);North South University(北南大学);University of Information Technology & Sciences(信息技术与科学大学)
MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions
Data and Web Science Group, University of Mannheim(曼海姆大学数据与网络科学小组);Max-Planck-Institute for Informatics, Saarland Informatics Campus(马克斯·普朗克研究所信息学研究所萨尔兰信息学校区)
本文提出DispBench基准测试工具,系统评估视差估计方法在对抗攻击和2D Common Corruptions等合成损坏下的鲁棒性,并通过迄今最广泛的分析揭示准确率、可靠性与泛化能力之间的关键相关性。
Adversarial Graph Neural Network Benchmarks: Towards Practical and Fair Evaluation
Department of Computer Science, University of Manitoba(曼尼托巴大学计算机科学系);University of Toledo(托莱多大学);NEC Laboratories Europe(NEC欧洲实验室);Department of Computer Science, Virginia Tech, USA(弗吉尼亚理工学院弗吉尼亚理工大学计算机科学系);AI Initiative - University of Central Florida(中央佛罗里达大学人工智能计划)
PRBench: A Standardized Probabilistic Robustness Benchmark
WMG, University of Warwick(沃里克大学WMG学院);Department of Computer Science, University of Liverpool(利物浦大学计算机科学系);College of Computer Science, Nankai University(南开大学计算机学院);School of Computing, National University of Singapore(新加坡国立大学计算学院)
Robust Physical-World Attacks on Deep Learning Models
University of Michigan(密歇根大学);University of Washington(华盛顿大学);University of California, Berkeley(加利福尼亚大学伯克利分校);Samsung Research America(三星美国研究院);Stony Brook University(石溪大学)