arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09349cs.LGcs.CRcs.CV
收藏

通过缩放定律与人类对齐研究探究对抗鲁棒性的极限

Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies

Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室)

Brian R. Bartoldson, James Diffenderfer, Konstantinos Parasyris, Bhavya Kailkhura

AI总结:

该研究构建首个对抗训练缩放定律,分析影响图像分类器对抗鲁棒性的因素,发现现有方法计算效率低,以更少算力提升了SOTA鲁棒性,同时预测鲁棒性会在90%左右达上限,人类表现也类似,并指出未来研究方向。

摘要:

本文重新探讨了一个研究已久却仍未解决的简单问题:如何让图像分类器对难以察觉的扰动具备鲁棒性。以CIFAR10为例,当前最优(SOTA)的干净准确率约为100%,但针对ℓ∞范数有界扰动的SOTA鲁棒性仅勉强超过70%。为理解这一差距,我们通过构建首个对抗训练缩放定律,分析了模型规模、数据集规模和合成数据质量对鲁棒性的影响。我们的缩放定律揭示了现有方法的低效性,并为推动该领域发展提供了可行的反馈。例如,我们发现SOTA方法明显偏离了计算最优配置,在其鲁棒性水平下使用了过多的计算量。利用计算高效的配置,我们以减少20%的训练浮点运算量(FLOPs)和70%的推理FLOPs超越了此前的SOTA。我们训练了多种计算高效的模型,其中最佳模型的AutoAttack准确率达到74%(提升了3%)。然而,我们的缩放定律也预测,鲁棒性增长缓慢,之后会在90%的水平趋于平稳:通过扩大规模来大幅超越我们的新SOTA是不切实际的,完美的鲁棒性也不可能实现。为更好地理解这一预测极限,我们对欺骗了我们最优模型的AutoAttack数据开展了小规模人类评估。令人担忧的是,我们估计人类的表现也在90%附近趋于平稳,我们证明这是由于ℓ∞约束的攻击生成了与原始标签不一致的无效图像。在明确了这些限制障碍后,我们概述了未来研究的可行方向。

英文摘要:

This paper revisits the simple, long-studied, yet still unsolved problem of making image classifiers robust to imperceptible perturbations. Taking CIFAR10 as an example, SOTA clean accuracy is about $100$%, but SOTA robustness to $\ell_{\infty}$-norm bounded perturbations barely exceeds $70$%. To understand this gap, we analyze how model size, dataset size, and synthetic data quality affect robustness by developing the first scaling laws for adversarial training. Our scaling laws reveal inefficiencies in prior art and provide actionable feedback to advance the field. For instance, we discovered that SOTA methods diverge notably from compute-optimal setups, using excess compute for their level of robustness. Leveraging a compute-efficient setup, we surpass the prior SOTA with $20$% ($70$%) fewer training (inference) FLOPs. We trained various compute-efficient models, with our best achieving $74$% AutoAttack accuracy ($+3$% gain). However, our scaling laws also predict robustness slowly grows then plateaus at $90$%: dwarfing our new SOTA by scaling is impractical, and perfect robustness is impossible. To better understand this predicted limit, we carry out a small-scale human evaluation on the AutoAttack data that fools our top-performing model. Concerningly, we estimate that human performance also plateaus near $90$%, which we show to be attributable to $\ell_{\infty}$-constrained attacks' generation of invalid images not consistent with their original labels. Having characterized limiting roadblocks, we outline promising paths for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑