词语的旅行：Transformer 的几何解释
Traveling Words: A Geometric Interpretation of Transformers
AI总结：
本文用几何视角解释 Transformer，将层归一化视为把特征约束到超球面，并通过 GPT-2 验证注意力模式，揭示词语语义沿超球面演化的机制。
摘要：
Transformer 已显著推动自然语言处理领域的发展，但理解其内部机制仍是一项挑战。本文提出一种新颖的几何视角，以阐明 Transformer 运算的内部机制。我们的主要贡献在于说明层归一化如何将潜在特征约束在一个超球面上，从而使注意力能够在该表面上塑造词语的语义表示。这一几何视角将迭代精炼和上下文嵌入等已有性质无缝地联系起来。我们通过探测一个预训练的 124M 参数 GPT-2 模型来验证这些见解。我们的发现揭示了早期层中清晰的 query-key 注意力模式，并在先前关于更深层注意力头具有主题特定性质的观察基础上进行了拓展。借助这些几何见解，我们提出了对 Transformer 的直观理解，将其描述为对词语粒子沿超球面运动轨迹进行建模的过程。
英文摘要：
Transformers have significantly advanced the field of natural language processing, but comprehending their internal mechanisms remains a challenge. In this paper, we introduce a novel geometric perspective that elucidates the inner mechanisms of transformer operations. Our primary contribution is illustrating how layer normalization confines the latent features to a hyper-sphere, subsequently enabling attention to mold the semantic representation of words on this surface. This geometric viewpoint seamlessly connects established properties such as iterative refinement and contextual embeddings. We validate our insights by probing a pre-trained 124M parameter GPT-2 model. Our findings reveal clear query-key attention patterns in early layers and build upon prior observations regarding the subject-specific nature of attention heads at deeper layers. Harnessing these geometric insights, we present an intuitive understanding of transformers, depicting them as processes that model the trajectory of word particles along the hyper-sphere.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
