Physical attacks against object detection have gained increasing attention due to their significant practical implications. However, conducting physical experiments is extremely time-consuming and labor-intensive. Moreover, physical dynamics and cross-domain transformation are challenging to strictly regulate in the real world, leading to unaligned evaluation and comparison, severely hindering the development of physically robust models. To accommodate these challenges, we explore utilizing realistic simulation to thoroughly and rigorously benchmark physical attacks with fairness under controlled physical dynamics and cross-domain transformation. This resolves the problem of capturing identical adversarial images that cannot be achieved in the real world. Our benchmark includes 20 physical attack methods, 48 object detectors, comprehensive physical dynamics, and evaluation metrics. We also provide end-to-end pipelines for dataset generation, detection, evaluation, and further analysis. In addition, we perform 8064 groups of evaluation based on our benchmark, which includes both overall evaluation and further detailed ablation studies for controlled physical dynamics. Through these experiments, we provide in-depth analyses of physical attack performance and physical adversarial robustness, draw valuable observations, and discuss potential directions for future research.
Codebase: https://github.com/JiaweiLian/Benchmarking_Physical_Attack
IGAff: Benchmarking Adversarial Iterative and Genetic Affine Algorithms on Deep Neural Networks
Faculty of Automatic Control；Computers, National University of Science；National Institute for Research \& Development in Informatics - ICI Bucharest, Bucharest, Romania
NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations
School of Computing, Wichita State University（威斯康星州立大学计算机学院）；Department of Computer Science, American International University-Bangladesh（孟加拉国国际大学计算机科学系）
MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics
MSU Institute for Artificial Intelligence（莫斯科大学人工智能研究所）；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；ISP RAS Research Center for Trusted Artificial Intelligence（ISP俄罗斯科学院可信人工智能研究中心）；Innopolis University（因诺波利斯大学）
Towards Benchmarking and Assessing Visual Naturalness of Physical World Adversarial Attacks
Beihang University（北京航空航天大学）；Tsinghua University（清华大学）；Zhongguancun Laboratory（中关村实验室）；Institute of data space, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）
Benchmarking Unified Face Attack Detection via Hierarchical Prompt Tuning
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)（多模态人工智能系统国家重点实验室（MAIS），自动化研究所，中国科学院（CASIA））；Department of Computer Science, College of Computing, City University of Hong Kong（计算机科学系，计算学院，香港城市大学）；School of Computer Science and Engineering, Faculty of Innovation Engineering, Macau University of Science and Technology（计算机科学与工程学院，创新工程学院，澳门科学理工学院）；Department of Computer Science and Engineering, Michigan State University（计算机科学与工程系，密歇根州立大学）；School of Cyber Science and Technology, University of Science and Technology of China（网络科学与技术学院，中国科学技术大学）；Ant Group（蚂蚁集团）；Imperial College London（伦敦帝国理工学院）
