通过字典学习实现Transformer可视化：将上下文嵌入作为Transformer因子的线性叠加
Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors
AI总结：
本文提出利用字典学习将Transformer的上下文嵌入表示为Transformer因子的线性叠加，通过可视化揭示其捕获的层次语义结构（如词级多义消歧、句级模式形成和长距离依赖），为理解Transformer网络内部机制提供了新见解。
摘要：
自引入以来，Transformer网络彻底改变了NLP表示学习。尽管已付出巨大努力来解释Transformer中的表示，但人们普遍认为我们的理解还不够。一个重要原因是缺乏足够的可视化工具来进行详细分析。在本文中，我们提出使用字典学习将这些“黑盒”打开为Transformer因子的线性叠加。通过可视化，我们展示了Transformer因子捕获的层次语义结构，例如词级多义消歧、句级模式形成和长距离依赖。其中一些模式证实了传统的先验语言学知识，而其余则相对出乎意料，这可能提供新的见解。我们希望该可视化工具能带来进一步的知识并更好地理解Transformer网络的工作原理。代码可在https://github.com/zeyuyun1/TransformerVis获取。
英文摘要：
Transformer networks have revolutionized NLP representation learning since they were introduced. Though a great effort has been made to explain the representation in transformers, it is widely recognized that our understanding is not sufficient. One important reason is that there lack enough visualization tools for detailed analysis. In this paper, we propose to use dictionary learning to open up these "black boxes" as linear superpositions of transformer factors. Through visualization, we demonstrate the hierarchical semantic structures captured by the transformer factors, e.g., word-level polysemy disambiguation, sentence-level pattern formation, and long-range dependency. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights. We hope this visualization tool can bring further knowledge and a better understanding of how transformer networks work. The code is available at https://github.com/zeyuyun1/TransformerVis
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
