Transformer前馈层是键值存储器
Transformer Feed-Forward Layers Are Key-Value Memories
摘要：
前馈层构成了Transformer模型参数的三分之二，但它们在网络中的作用仍未得到充分研究。我们发现，在基于transformer的语言模型中的前馈层作为键值记忆，其中每个键与训练样本中的文本模式相关，并且每个值引起输出词汇的分布。我们的实验表明，学习到的模式是人类可解释的，并且较低的层倾向于捕获浅模式，而上层学习更多的语义。这些值通过诱导输出分布来补充键的输入模式，输出分布将概率质量集中在可能紧接在每个模式之后出现的标记上，特别是在上层。最后，我们证明了前馈层的输出是其记忆的组成，随后通过残差连接在模型的各个层中进行细化，以产生最终的输出分布。
英文摘要：
Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys' input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model's layers via residual connections to produce the final output distribution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
