头部假设：理解BERT中多头注意的统一统计方法
The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT
摘要：
多头注意力头是基于变形金刚的模型中的中流砥柱。已经提出了不同的方法来分类的基础上的令牌之间的关系，具有高的成对注意力的每个注意头的作用。这些角色包括句法（具有某种句法关系的标记）、局部（邻近标记）、块（同一句子中的标记）和块（特殊的[CLS]、[SEP]标记）。现有的分类方法存在两个主要挑战：（a）没有跨研究或跨功能角色的标准分数，以及（b）这些分数通常是跨句子测量的平均数量，而没有捕获统计意义。在这项工作中，我们正式提出了一个简单而有效的分数，概括了所有的角色的注意头，并采用假设检验这个分数的强大的推理。这为我们提供了正确的镜头来系统地分析注意力头，并自信地评论分析BERT模型的许多常见问题。特别是，我们评论在同一个注意头的多个功能角色的协同定位，跨层的注意头的分布，以及微调特定的NLP任务对这些功能角色的影响。
英文摘要：
Multi-headed attention heads are a mainstay in transformer-based models. Different methods have been proposed to classify the role of each attention head based on the relations between tokens which have high pair-wise attention. These roles include syntactic (tokens with some syntactic relation), local (nearby tokens), block (tokens in the same sentence) and delimiter (the special [CLS], [SEP] tokens). There are two main challenges with existing methods for classification: (a) there are no standard scores across studies or across functional roles, and (b) these scores are often average quantities measured across sentences without capturing statistical significance. In this work, we formalize a simple yet effective score that generalizes to all the roles of attention heads and employs hypothesis testing on this score for robust inference. This provides us the right lens to systematically analyze attention heads and confidently comment on many commonly posed questions on analyzing the BERT model. In particular, we comment on the co-location of multiple functional roles in the same attention head, the distribution of attention heads across layers, and effect of fine-tuning for specific NLP tasks on these functional roles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
