arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.03286cs.CL
收藏

GlobEnc:通过在Transformers中扩展整个编码器层来量化全局Token属性

GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers

University of Tehran(德黑兰大学);Tehran Institute for Advanced Studies, Khatam University(哈塔姆大学德黑兰高等研究院);Iran University of Science and Technology(伊朗科技大学)

Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar

摘要:

人们对解释Transformers的潜在动力学越来越感兴趣。虽然自我注意力模式最初被认为是主要的选择,但最近的研究表明,整合其他成分可以产生更准确的解释。本文介绍了一种新的令牌属性分析方法,该方法将编码器块中的所有组件合并在一起,并在整个层中聚合。通过大量的定量和定性实验,我们证明了我们的方法可以产生忠实和有意义的全球令牌属性。我们的实验表明,将几乎每个编码器组件都集成在一起,可以在局部(单层)和全局(整个模型)设置中进行越来越准确的分析。我们的全球归因分析显着优于以前的方法在各种任务的相关性与基于梯度的显着性分数。我们的代码可以在https://github.com/mohsenfayyaz/GlobEnc上免费获得。

英文摘要:

There has been a growing interest in interpreting the underlying dynamics of Transformers. While self-attention patterns were initially deemed as the primary option, recent studies have shown that integrating other components can yield more accurate explanations. This paper introduces a novel token attribution analysis method that incorporates all the components in the encoder block and aggregates this throughout layers. Through extensive quantitative and qualitative experiments, we demonstrate that our method can produce faithful and meaningful global token attributions. Our experiments reveal that incorporating almost every encoder component results in increasingly more accurate analysis in both local (single layer) and global (the whole model) settings. Our global attribution analysis significantly outperforms previous methods on various tasks regarding correlation with gradient-based saliency scores. Our code is freely available at https://github.com/mohsenfayyaz/GlobEnc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑