你的基准真的具有对抗性吗？AdvScore：评估以人类为基础的对抗性
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness
AI总结：
本文提出以人类为基础的对抗性评估指标AdvScore，用于识别劣质样本并衡量数据集是否仍具挑战，并据此构建高质量对抗QA数据集AdvQA，结合9,347条人类回答与十个模型预测追踪五年模型演进。
摘要：
对抗性数据集应当通过提供人类表现良好而模型表现不佳的样本来验证AI的鲁棒性。然而，随着模型不断演进，数据集可能会变得过时。由于缺乏衡量对抗性的标准化指标，衡量一个数据集是否仍具有对抗性受到阻碍。我们提出AdvScore，这是一种以人类为基础的评估指标，通过捕捉模型和人类的不同能力，同时识别劣质样本来评估数据集的对抗性。随后，我们利用AdvScore推动一种新的数据集创建流程，以生成真实且高质量的对抗样本，从而使我们能够收集一个对抗性问答（QA）数据集AdvQA。我们使用9,347条人类回答和十个语言模型的预测来应用AdvScore，追踪模型从2020年到2024年这五年间的改进。因此，AdvScore为实现与人类能力相当的鲁棒性提供了指导。此外，它有助于确定对抗性数据集在多大程度上持续构成挑战，确保这些数据集有效测试模型能力，而不是反映过时或过于人工化的困难。
英文摘要：
Adversarial datasets should validate AI robustness by providing samples on which humans perform well, but models do not. However, as models evolve, datasets can become obsolete. Measuring whether a dataset remains adversarial is hindered by the lack of a standardized metric for measuring adversarialness. We propose AdvScore, a human-grounded evaluation metric that assesses a dataset's adversarialness by capturing models' and humans' varying abilities while also identifying poor examples. We then use AdvScore to motivate a new dataset creation pipeline for realistic and high-quality adversarial samples, enabling us to collect an adversarial question answering (QA) dataset, AdvQA. We apply AdvScore using 9,347 human responses and ten language models' predictions to track model improvement over five years, from 2020 to 2024. AdvScore thus provides guidance for achieving robustness comparable with human capabilities. Furthermore, it helps determine to what extent adversarial datasets continue to pose challenges, ensuring that, rather than reflecting outdated or overly artificial difficulties, they effectively test model capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
