AI 中文总结
该研究针对Transformer可解释性研究中表示演变分析的不足，提出基于持续同调的Transformer几何观测站TGO-IV拓扑框架，通过多种拓扑工具全面分析表示点云的全局拓扑演变。
AI 中文摘要
Transformer在语言处理和计算机视觉领域产生了深远影响。随着解答“Transformer如何学习”这一关键问题的研究不断增多，现有可解释性研究主要分析孤立层的表示或整个网络，而各表示的发展演变及其在Transformer各层中的流形仍未得到充分探索。本研究旨在对表示的演变进行全面分析，即表示点云在各层间的变换过程，进而尝试分离出各层或建立一种趋势，以更接近解释原始输入表示如何以及何时演变为任务相关的特征表示。因此，Transformer几何观测站TGO-IV引入了一种基于持续同调的拓扑框架，用于分析Transformer表示的演变。TGO-IV并非仅研究局部几何属性，而是从 token 级表示点云构建Vietoris-Rips单纯复形，并研究其持续拓扑特征在Transformer各层中的演变。该框架包含互补的拓扑观测工具，包括持续图、条形码图、贝蒂曲线、持续景观、瓶颈距离和Wasserstein距离，可全面分析表示点云的全局拓扑在整个前向传播过程中的发展情况。
英文摘要
Transformers have had a profound impact on the world of language processing and computer vision. As efforts to answer the million-dollar question of ``How does a Transformer learn?" have been increasing, existing interpretability studies primarily analyze representations at isolated layers or the network as a whole, while the developmental evolution of individual representations and its manifolds across transformer layers remains underexplored. With this work, we aim at providing a comprehensive analysis of the evolution of representations as the representation point cloud transforms across the layers; thereby attempting to isolate layers or establish a trend which comes closer to justifying how and when raw input representations evolve into task-relevant feature representations. Thus, Transformer Geometry Observatory-TGO-IV introduces a topological framework for analysing the evolution of Transformer representations through the lens of Persistent Homology. Rather than studying local geometric properties alone, TGO-IV constructs Vietoris--Rips simplicial complexes from token-level representation point clouds and investigates the evolution of their persistent topological signatures across Transformer layers. The proposed framework comprises complementary topological observatories including Persistence Diagrams, Barcode Diagrams, Betti Curves, Persistence Landscapes, Bottleneck Distance, and Wasserstein Distance, enabling a comprehensive analysis of how the global topology of representation point clouds develops throughout the forward pass.
