Adversarial GLUE:一个用于语言模型鲁棒性评估的多任务基准
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
摘要:
大规模预训练的语言模型在广泛的自然语言理解(NLU)任务中取得了巨大成功,甚至超越了人类的表现。然而,最近的研究表明,这些模型的鲁棒性可能会受到精心制作的文本对抗性示例的挑战。虽然已经提出了几个单独的数据集来评估模型的鲁棒性,但仍然缺少一个原则性和全面的基准。在本文中,我们提出了Adversarial GLUE(AdvGLUE),一个新的多任务基准,定量和彻底地探索和评估现代大规模语言模型在各种类型的对抗攻击下的脆弱性。特别是,我们系统地将14种文本对抗攻击方法应用于GLUE任务,以构建AdvGLUE,并由人类进一步验证其可靠的注释。我们的研究结果总结如下。(i)现有的大多数对抗攻击算法都容易产生无效或模糊的对抗示例,其中约90%的攻击算法要么改变了原始语义,要么误导了人类注释者。因此,我们执行仔细的过滤过程,以策划高质量的基准。(ii)我们测试的所有语言模型和健壮的训练方法在AdvGLUE上的表现都很差,分数远远落后于良性的准确性。我们希望我们的工作将推动开发更隐蔽和语义保护的新的对抗性攻击,以及针对复杂对抗攻击的新的健壮语言模型。AdvGLUE可在https://adversarialglue.github.io上获得。
英文摘要:
Large-scale pre-trained language models have achieved tremendous success across a wide range of natural language understanding (NLU) tasks, even surpassing human performance. However, recent studies reveal that the robustness of these models can be challenged by carefully crafted textual adversarial examples. While several individual datasets have been proposed to evaluate model robustness, a principled and comprehensive benchmark is still missing. In this paper, we present Adversarial GLUE (AdvGLUE), a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Our findings are summarized as follows. (i) Most existing adversarial attack algorithms are prone to generating invalid or ambiguous adversarial examples, with around 90% of them either changing the original semantic meanings or misleading human annotators as well. Therefore, we perform a careful filtering process to curate a high-quality benchmark. (ii) All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy. We hope our work will motivate the development of new adversarial attacks that are more stealthy and semantic-preserving, as well as new robust language models against sophisticated adversarial attacks. AdvGLUE is available at https://adversarialglue.github.io.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。