使用NDCG和倒数排名指标评估神经网络防御与攻击
Evaluation of Neural Networks Defenses and Attacks using NDCG and Reciprocal Rank Metrics
AI总结：
本文针对对抗样本攻击与防御评估中传统分类指标区分度不足的问题，借鉴信息检索中的NDCG和倒数排名指标，提出了两种适用于多类分类任务的新颖评估指标，并在VGG19模型和ImageNet数据集上验证了其卓越的信息量和区分度。
摘要：
通过输入修改（即对抗样本）攻击神经网络的问题近期引起了广泛关注。这些攻击相对容易生成且难以检测，构成了许多建议的防御试图缓解的安全漏洞。然而，攻击和防御效果的评估通常依赖于传统的分类指标，而没有充分适应对抗场景。这些指标大多基于准确率，因此可能具有有限的适用范围和较低的区分能力。其他指标不考虑神经网络功能的独特特征，或间接测量攻击的影响（例如，通过其生成的复杂性）。在本文中，我们提出了两个专门设计的指标，用于测量多类分类任务中攻击或防御恢复效果对神经网络输出的影响。受信息检索文献中使用的归一化折损累计增益（NDCG）和倒数排名指标的启发，我们将神经网络预测视为结果的排序列表。利用关于排名概率的额外信息，使我们能够定义适合当前任务的新颖指标。我们使用各种攻击和防御在预训练的VGG19模型和ImageNet数据集上评估了我们的指标。与常见的分类指标相比，我们提出的指标表现出卓越的信息量和区分度。
英文摘要：
The problem of attacks on neural networks through input modification (i.e., adversarial examples) has attracted much attention recently. Being relatively easy to generate and hard to detect, these attacks pose a security breach that many suggested defenses try to mitigate. However, the evaluation of the effect of attacks and defenses commonly relies on traditional classification metrics, without adequate adaptation to adversarial scenarios. Most of these metrics are accuracy-based, and therefore may have a limited scope and low distinctive power. Other metrics do not consider the unique characteristics of neural networks functionality, or measure the effect of the attacks indirectly (e.g., through the complexity of their generation). In this paper, we present two metrics which are specifically designed to measure the effect of attacks, or the recovery effect of defenses, on the output of neural networks in multiclass classification tasks. Inspired by the normalized discounted cumulative gain and the reciprocal rank metrics used in information retrieval literature, we treat the neural network predictions as ranked lists of results. Using additional information about the probability of the rank enabled us to define novel metrics that are suited to the task at hand. We evaluate our metrics using various attacks and defenses on a pretrained VGG19 model and the ImageNet dataset. Compared to the common classification metrics, our proposed metrics demonstrate superior informativeness and distinctiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
