发表机构
Fraunhofer SIT（弗劳恩霍夫研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究对抗鲁棒性模型与普通模型的精度差距，引入VanillaBench基准，计算多个对抗训练模型与普通参考模型的精度差异，发现鲁棒性-精度权衡比通常认为的大，建议未来评估应报告普通参考精度差距。
AI 中文摘要
在过去十年中，对抗鲁棒性研究产生了数百个防御模型，但文献几乎普遍孤立地报告鲁棒性结果：展示了鲁棒模型的标准（干净）精度和对抗精度，但很少量化与相应的普通模型的差距。我们引入了VanillaBench，这是一个使这种差距明确的系统基准。对于RobustBench在四个威胁模型中编目的每个对抗训练模型，我们计算了与来自Papers with Code的多个普通参考模型的精度差异，这些差异是在所有条目和无额外数据条目、鲁棒模型发表年份的最佳普通模型以及架构匹配基线的基础上计算的。在所有186个鲁棒模型中，相对于最佳普通模型的平均干净精度下降范围为-7.7至-29.5个百分点，即使每个轨迹中最鲁棒的单个模型仍比其同期普通对应模型落后4.0至21.0个百分点。架构匹配比较揭示了平均差距为-3.5至-17.5个百分点。将这种架构匹配比较限制在具有相同架构的已知普通精度的模型上，而不是从相关架构近似得出，将差距缩小到-4.0至-14.0个百分点。这些结果表明，鲁棒性-精度权衡比个别论文通常传达的要大得多。此信息对从业者和决策者至关重要。在实际环境中部署模型时，鲁棒性的精度成本直接影响业务成果，但当前出版物未提供评估所需的普通基线。我们认为未来的鲁棒性评估应将普通参考精度差距作为标准组成部分报告。
英文摘要
Adversarial robustness research has produced hundreds of defended models over the past decade, yet the literature almost universally reports robustness results in isolation: standard (clean) accuracy and adversarial accuracy of the robust model are shown, but the gap to the corresponding vanilla model is rarely quantified. We introduce VanillaBench, a systematic benchmark that makes this gap explicit. For every adversarially-trained model catalogued by RobustBench across four threat models, we compute the accuracy difference against multiple vanilla references from Papers with Code, computed over both all entries and no-extra-data entries, the best vanilla model as of the robust model's publication year, and an architecture-matched baseline. Across all 186 robust models, the mean delta clean relative to the best vanilla model ranges from -7.7 to -29.5 percentage points, and even the single most robust model per track still trails its temporal vanilla counterpart by 4.0-21.0 points. The architecture-matched comparison, which isolates the effect of adversarial training from architectural differences, reveals a mean gap of -3.5 to -17.5 points. Restricting this architecture-matched comparison to models whose vanilla accuracy is known for the exact same architecture, rather than approximated from a related one, narrows the gap to -4.0 to -14.0 points. These results demonstrate that the robustness-accuracy trade-off is substantially larger than what is typically conveyed by individual papers. This information is critical for practitioners and decision-makers. When deploying models in real-world settings, the accuracy cost of robustness directly affects business outcomes, yet current publications do not provide the vanilla baseline needed to assess it. We argue that future robustness evaluations should report vanilla-referenced accuracy gaps as a standard component.
