Transformer几何观测站TGO-I：谱几何观测站
Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory
- Department of Electronics Engineering Sardar Vallabhai National Institute of Technology (SVNIT), Surat, India（电子工程系萨达尔·瓦拉巴伊国家理工学院（SVNIT），印度萨拉特）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出TGO框架，通过分析ViT表示的谱几何（有效秩、稳定秩、参与比、谱熵、谱平坦度、谱各向异性等），发现训练过程中维度利用增加、各向异性降低、谱熵和参与比上升，最终CLS标记表示具有最高有效维度和最低各向异性。
AI中文摘要：
尽管Vision Transformers（ViTs）被广泛采用并在众多计算机视觉应用中取得成功，对其维度和表示几何的基本理解仍然相对未被充分探索。为了弥补这一差距，我们引入了Transformer几何观测站（TGO），这是一个系统的实验和分析流程框架，旨在研究Vision Transformers的表示几何和动态。TGO-I是该框架的第一部分，专注于ViT表示的谱几何。使用在ImageNet-100上训练的ViT-Small/16模型，我们分析了训练过程中的有效秩、稳定秩、参与比、谱熵、谱平坦度、谱各向异性、协方差结构、特征谱和奇异值谱。我们的结果揭示了维度利用的一致增加，伴随着各向异性降低、谱熵增加、参与比增加以及逐渐平坦的特征谱。与常见的直觉（即训练应将信息集中到少数主导方向）相反，我们观察到方差在表示维度上的逐渐重新分布。这一现象在最终的CLS标记表示中尤为明显，该表示在网络中表现出最高的有效维度和最低的各向异性。
英文摘要：
Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers. TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations. Using a ViT-Small/16 model trained on ImageNet-100, we analyze Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra throughout training. Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra. Contrary to the common intuition that training should concentrate information into a small number of dominant directions, we observe a progressive redistribution of variance across representational dimensions. This phenomenon is particularly pronounced in the final CLS token representation, which exhibits the highest effective dimensionality and lowest anisotropy within the network.
