基于谱能量保持的规范性SVD启发注意力
Prescriptive SVD-Inspired Attention via Spectral Energy Retention
浏览论文内容
中文总结 AI 辅助
本文提出诊断-干预-验证框架，评估SVDA中谱能量保持干预，在多个数据集上有效减少计算量且精度损失极小，支持其作为可解释注意力机制。
中文摘要 AI 辅助
自注意力是现代Transformer架构的核心，但其密集的点积公式使得难以识别哪些内部方向在结构上重要，哪些可以在不破坏模型的情况下进行修改。SVD启发注意力（SVDA）通过将学习到的对角谱引入查询-键分数交互中，部分解决了这一问题，使潜在的注意力方向通过谱熵、有效秩、稀疏性、对齐性、选择性和扰动响应等指标变得明确可检查。本文探讨了从诊断性解释到操作性干预的转变。提出了一个诊断-干预-验证框架，并评估了一种干预措施：注意力分数路径中的谱能量保持。在FashionMNIST、CIFAR-10、CIFAR-100和Food-101上，ρ=0.90的处方移除了24.5%至53.7%的分数方向，参数减少了2.6%至4.3%，估计的MACs减少了2.8%至5.4%。降维模型在三个种子上的配对平均准确率变化范围为-0.03至+0.05个百分点。这些结果支持SVDA作为一种内在可解释的注意力机制，其学习到的谱暴露了一个用于确定性和可验证地修改注意力分数形成的操作坐标系。
英文摘要
Self-attention is central to modern Transformer architectures, but its dense dot-product formulation makes it difficult to identify which internal directions are structurally important and which can be modified without disrupting the model. SVD-Inspired Attention (SVDA) addresses part of this problem by introducing a learned diagonal spectrum into the query-key score interaction, making latent attention directions explicitly inspectable through indicators such as spectral entropy, effective rank, sparsity, alignment, selectivity, and perturbation response. This paper examines the transition from diagnostic interpretation to operational intervention. A diagnosis--intervention--verification framework is proposed, and one intervention is evaluated: spectral energy retention in the attention-score pathway. Across FashionMNIST, CIFAR-10, CIFAR-100, and Food-101, the $ρ=0.90$ prescription removes 24.5--53.7\% of score directions, reduces parameters by 2.6--4.3\%, and reduces estimated MACs by 2.8--5.4\%. The paired mean accuracy change of the dimension-reduced model ranges from $-0.03$ to $+0.05$ percentage points over three seeds. These results support SVDA as an intrinsically interpretable attention mechanism whose learned spectrum exposes an operational coordinate system for deterministic and verifiable modification of attention-score formation.
发表机构
- Athena Research Center（雅典娜研究中心）
机构由 AI 辅助整理，请以论文原文为准。
