Transformer 在近视方法下不可解释:以有界 Dyck 文法为例的研究
Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars
AI总结:
本文以有界 Dyck 文法学习任务为例,通过理论证明和实验表明 Transformer 的最优解在注意力模式上可近乎随机化,从而说明仅检查单个注意力头或权重矩阵的近视可解释性方法不可靠。
摘要:
可解释性方法旨在通过检查模型的各个方面(如权重矩阵或注意力模式)来理解训练模型(例如 Transformer)所实现的算法。在本工作中,我们结合理论结果与在合成数据上精心控制的实验,对那些只关注模型个别部分而非将网络作为整体考虑的方法提出批判性观点。我们考虑一个学习(有界)Dyck 语言的简单合成设置。理论上,我们证明(精确或近似)解决该任务的模型集合满足一种源自形式语言思想(泵引理)的结构化特征。我们利用这一特征表明最优解集合在性质上是丰富的;特别地,单层的注意力模式可以“近乎随机化”,同时保持网络的功能。我们还通过大量实验表明,这些构造不仅仅是理论上的产物:即使对模型架构施加严格限制,通过标准训练仍可得到截然不同的解。因此,基于检查 Transformer 中单个注意力头或权重矩阵的可解释性结论可能具有误导性。
英文摘要:
Interpretability methods aim to understand the algorithm implemented by a trained model (e.g., a Transofmer) by examining various aspects of the model, such as the weight matrices or the attention patterns. In this work, through a combination of theoretical results and carefully controlled experiments on synthetic data, we take a critical view of methods that exclusively focus on individual parts of the model, rather than consider the network as a whole. We consider a simple synthetic setup of learning a (bounded) Dyck language. Theoretically, we show that the set of models that (exactly or approximately) solve this task satisfy a structural characterization derived from ideas in formal languages (the pumping lemma). We use this characterization to show that the set of optima is qualitatively rich; in particular, the attention pattern of a single layer can be ``nearly randomized'', while preserving the functionality of the network. We also show via extensive experiments that these constructions are not merely a theoretical artifact: even after severely constraining the architecture of the model, vastly different solutions can be reached via standard training. Thus, interpretability claims based on inspecting individual heads or weight matrices in the Transformer can be misleading.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。