Transformer几何观测站TGO-III：语义几何观测站
Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory
浏览论文内容
中文总结 AI 辅助
本研究提出TGO-III框架分析ViT-Small/16的表征演化，揭示类别表征的几何变化，为语义扩展假说提供实证，扩展了Transformer几何观测站框架。
中文摘要 AI 辅助
随着视觉Transformer在现代AI中的广泛应用，分析其内在表征行为的需求日益重要。现有多数研究侧重token几何与训练动态，而表征协方差结构的演化及类别级几何组织却相对未被充分探索。本研究通过TGO-III（语义几何观测站，Semantic Geometry Observatory）探究ViT-Small/16各层表征演化过程中的语义几何与类别可分性，该框架旨在分析训练过程中语义组织、特征演化及类别表征几何的出现，采用线性探测准确率、Fisher比率、类别质心距离、局部本征维数、局部PCA秩等多个互补观测站，量化判别性表征的渐进演化。分析显示，类别表征的线性可分性逐步提升，Fisher判别力增强，类别质心间距增大，局部表征流形呈现结构化的类别依赖几何复杂性。这些观察为语义扩展假说提供了实证依据，表明此前观测到的流形扩张伴随表征逐步组织为更具判别性的语义结构。总体而言，TGO-III通过建立Transformer训练期间流形几何、协方差演化与语义组织的直接联系，扩展了Transformer几何观测站框架。
英文摘要
With the widespread adoption of Vision Transformers in modern AI, the need to analyze their inherent representational behavior has become increasingly important. While most existing studies emphasize token geometries and training dynamics, the evolution of representational covariance structures and class-level geometric organization remains comparatively underexplored. In this work, we investigate semantic geometry and class separability as representations evolve across the layers of ViT-Small/16 through TGO-III: Semantic Geometry Observatory. It is a framework designed to analyze the emergence of semantic organization, feature evolution, and class-wise representation geometry throughout training. The framework employs multiple complementary observatories, including Linear Probe Accuracy, Fisher Ratio, Class Centroid Distances, Local Intrinsic Dimension, and Local PCA Rank, to quantify the progressive evolution of discriminative representations. Our analysis reveals that class representations become progressively more linearly separable, Fisher discriminability increases, class centroids move farther apart, and local representation manifolds exhibit structured class-dependent geometric complexity. These observations provide empirical evidence supporting the Semantic Expansion Hypothesis, suggesting that the manifold expansion observed in previous observatories is accompanied by the progressive organization of representations into increasingly discriminative semantic structures. Collectively, TGO-III extends the Transformer Geometry Observatory framework by establishing a direct connection between manifold geometry, covariance evolution, and semantic organization during Transformer training.
