如何剖析布偶:Transformer嵌入空间的结构
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces
摘要:
基于Transformer架构的预训练嵌入已经在NLP社区掀起了风暴。我们表明,它们可以在数学上被重构为矢量因子的总和,并展示如何使用这种重构来研究每个组件的影响。我们提供的证据表明,多头注意力和前馈在所有下游应用中并不同样有用,以及微调对整个嵌入空间的影响的定量概述。这种方法使我们能够与以前的广泛研究建立联系,从向量空间各向异性到注意力权重。
英文摘要:
Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm. We show that they can mathematically be reframed as a sum of vector factors and showcase how to use this reframing to study the impact of each component. We provide evidence that multi-head attentions and feed-forwards are not equally useful in all downstream applications, as well as a quantitative overview of the effects of finetuning on the overall embedding space. This approach allows us to draw connections to a wide range of previous studies, from vector space anisotropy to attention weights.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。