The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input. Prior work strives to attribute model decisions to individual input features with different saliency measures, but they fail to explain how these input features interact with each other to reach predictions. In this paper, we propose a self-attention attribution method to interpret the information interactions inside Transformer. We take BERT as an example to conduct extensive studies. Firstly, we apply self-attention attribution to identify the important attention heads, while others can be pruned with marginal performance degradation. Furthermore, we extract the most salient dependencies in each layer to construct an attribution tree, which reveals the hierarchical interactions inside Transformer. Finally, we show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
Department of Computer & Information Science & Engineering, University of Florida(佛罗里达大学计算机与信息科学与工程系);School of Computer Science, University of Oklahoma(俄克拉荷马大学计算机科学学院);Department of Electrical Engineering and Computer Science, United States Military Academy(美国军事学院电子工程与计算机科学系)
The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models
Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);National University of Singapore(新加坡国立大学);Microsoft Research(微软研究院);IBM Research(IBM研究院)
Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Hunan Normal University(湖南师范大学)
Interpreto: An Explainability Library for Transformers
IRT Saint Exupéry Toulouse(伊尔杜夫圣埃克苏佩里图卢斯);IRIT Toulouse(图卢兹IRIT);Khoury College of Computer Sciences(科赫里计算机科学学院);Ampere(阿姆佩尔);MICS, CentraleSupélec(MICS,中央超导学院);Scienta Lab(科学实验室);Thales Avionics(泰勒斯航空电子);ANITI