arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.07556cs.CV
收藏

视觉变换器中的深度动态模式分解

Dynamic Mode Decomposition along Depth in Vision Transformers

NYU Tandon & NYU Abu Dhabi(纽约大学Tandon学院及阿布扎赫尔纽约大学)

Nishant Suresh Aswani, Saif Eddin Jabari

AI总结:

研究视觉变换器深度是否实现近似自主线性动力学,通过动态模式分解验证并分析其在不同深度下的稳定性与可扩展性。

摘要:

近期研究表明,连续视觉变换器(ViT)块(a)可以被线性映射替代,并(b)组织成递归计算阶段。我们询问这些观察是否一致:视觉变换器的深度是否实现了近似的自主线性动力学,允许一个单一的算子$K$在连续范围内递归应用?我们通过动态模式分解(DMD)测试这一假设,该方法从选定的连续隐藏状态对中拟合$K$,并通过$K^p$预测$p$步 ahead。在四个预训练的DINO ViT上,我们研究了实现稳定拟合所需的正则化、秩和校准预算。对于短跨度($p \leq 4$),$K^p$在DINOv3-H/16+上能够将无约束端点映射的余弦相似度控制在0.02以内,同时在每个跳过的块中恢复中间激活。在早期剪切开始时,拟合的算子压缩到秩$\ll d$,且需要最少的校准数据,而在跨token时,\texttt{cls}最易于线性化;这两个属性随着深度单调衰减。然而,这种局部保真度并不能迁移到下游任务。在最终隐藏状态中,经过剩余块的传播后,身份基线变得具有竞争力。

英文摘要:

Recent work has shown that contiguous vision transformer (ViT) blocks (a) can be replaced by a linear map and (b) organize into recurrent phases of computation. We ask whether these observations coincide: does ViT depth implement approximately \textit{autonomous linear} dynamics, admitting a single operator $K$ applied recurrently across a contiguous span? We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs and predicts $p$ steps ahead via $K^p$. On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting. For short spans ($p \leq 4$), $K^p$ tracks an unconstrained endpoint map to within $0.02$ cosine similarity on DINOv3-H/16+, while also recovering intermediate activations at each skipped block. At early cut starts, the fitted operators compress to rank $\ll d$ with minimal calibration data, and across tokens, \texttt{cls} is most amenable to linearization; both properties decay monotonically with depth. Yet this local fidelity does not transfer downstream. At the final hidden state, after propagating through the remaining blocks, an identity baseline becomes competitive.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑