发表机构
University of Tartu; Institute of Computer Science（塔尔图大学; 计算机科学研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究通过可视化训练过程的双维度特征演化，利用稀疏自编码器分析视觉Transformer的特征迁移规律，为理解其学习演化机制提供了工具。
AI 中文摘要
我们通过在网络深度（层）和训练时间（轮次）两个维度上可视化训练过程，为视觉Transformer（ViT）的特征演化提供了一种新视角。我们采用稀疏自编码器（SAE）从<[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]>token表示中提取候选稀疏特征，并比较它们在轮次-层对中的激活分布，这使我们能够研究特征层面的动态，而这些动态无法从表示层面的相似性度量中直接观察到。此外，我们证明该特征演化框架可用于描述特征迁移——即训练过程中特征最易被检测到的层发生变化的现象。实验表明，迁移集中在训练早期，且更常发生在较浅层而非深层，随着特征组织趋于稳定而减少；我们还发现深层比浅层更早且更强烈地稳定下来。结果表明，我们的方法可作为理解ViT如何学习和演化的工具。
英文摘要
We present a novel view on feature evolution in Vision Transformers (ViTs) by visualizing the training process over two dimensions -- network depth (layer) and training time (epochs). We employ Sparse Autoencoders (SAEs) to extract candidate sparse features from CLS-token representations and compare their activation profiles across epoch--layer pairs. This allows us to study feature-level dynamics that are not directly visible from representation-level similarity measures. Furthermore, we demonstrate how this framework of feature evolution allows us to describe feature migration, the change in the layer where a feature is most detectable during training. Our experiments show that migration is concentrated early in training, occurs more often toward earlier layers than toward deeper layers, and declines as feature organization stabilizes. We further find that deeper layers stabilize earlier and more strongly than shallow layers. The results show that our approach can be employed as a tool for understanding how ViTs learn and evolve.
CommentsAccepted to CIKM 2026
