重新思考具有线性时不变动态的MAE
Learning Dynamic Evidence Routes for Vision Transformer Probing
AI总结：
本文提出SSMProbe框架，利用状态空间模型探索视觉表示中序列顺序的重要性，发现预训练目标影响token结构，不同模型在顺序依赖性上表现各异。
摘要：
标准的视觉模型表示探针依赖于数学上排列不变的操作，如全局平均池化（GAP）或CLS标记，将补丁表示视为无结构的词袋。我们通过证明在冻结的视觉表示中（例如MAE、BEiT、DINOv2和ViT作为CLS消融极端）token顺序是一个关键且可利用的维度来挑战这一范式。我们提出了SSMProbe，一个由状态空间模型（SSM）驱动的探针框架。作为离散的线性时不变（LTI）动力系统运行，SSM在排列敏感的探针中起作用，由于固有的记忆衰减，序列顺序严格决定最终状态。将token顺序 formulation 为信息调度问题，我们比较了固定扫描启发式方法与从下游监督中学习的可微软排列（基于Sinkhorn）。在标准和细粒度分类基准上的评估揭示了显著的顺序差距：虽然固定扫描在高度局部化的补丁特征上表现惨淡，我们的学习软排列却能从原本高度局部化的补丁序列中提取出高度竞争性的性能。我们发现预训练目标从根本上塑造了token结构：DINOv2将全局语义集中在优化的CLS标记中，使补丁超专业化，纯MAE保持分布表示，具有异质补丁信息量，而ViT代表了监督CLS主导的极端。BEiT处于中间位置。这种异质性是顺序依赖的——意味着SSM探针的性能严重依赖于哪些token放在哪个时间位置——而不仅仅是空间网格的拓扑属性。SSMProbe学习的路由有效地发现了并利用了这种异质性，提供了一种强大的新诊断透镜用于视觉表示分析。
英文摘要：
Probing frozen vision transformers typically uses permutation-invariant aggregation (GAP or $\texttt{[CLS]}$), treating patch tokens as an unstructured set. Content-dependent probes such as self-attention are useful accuracy controls, but they do not expose a fixed token schedule or fixed position weights for auditing. We introduce $\textbf{SSMProbe}$, an explicitly inspectable probe that replaces invariant pooling with a Sinkhorn-learned evidence route followed by a diagonal S4 decoder. The S4 decoder is a linear time-invariant (LTI) system whose final state has fixed, position-dependent coefficients, so the probe-induced routed sequence can be audited as a concrete object rather than inferred only from accuracy. Our central measurement is the geometry of routed evidence: which patch tokens are moved to influential positions by this diagnostic, whether those tokens form spatially organized regions or random-like dispersed sets, and how the fixed S4 kernel weights them. Across MAE, BEiT, DINOv2, and supervised ViT, this route geometry separates MAE's dispersed, nearly random-like routes from the more spatially organized routes of BEiT, ViT, and DINOv2, with DINOv2 retaining a distinct strong $\texttt{[CLS]}$ profile. SSMProbe uses the mathematical transparency of state-space models to turn a frozen ViT readout into an auditable evidence-routing analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
