从RoPE导数得到的精确有限注意力响应

中文摘要 AI 辅助

我们推导了注意力干预的精确局部响应，使得候选编辑可以通过缓存的基线和一次反向传播进行评分。出发点是RoPE导数$\partial_p z(p) = A z(p)$：其积分给出有限位置位移，我们将其贯穿softmax而不对旋转或归一化进行线性化。由此产生的预测在768个保留提示集上执行的92,160次位置编辑中达到95.36--96.52%的符号准确率，相对于位置雅可比矩阵将答案边际平均绝对误差（MAE）降低了73.6--82.5%，相对于零基线降低了36.2--50.9%。对于同时的键和值编辑，相同的均差演算分离出交互项$C_{KV} = \sum_j (p'_j - p_j)\\,\varepsilon_j$，而单独归因相加会遗漏该项。保留该项在跨越两个Qwen规模、两个任务和多个层的5,120次干预扫描的每个设置中，将下游边际MAE降低了超过九倍；相对于二次交互校正，降低幅度为75.9--98.5%。精确性涉及编辑后的注意力写入；下游预测将该响应与基线梯度进行收缩，并通过原生执行进行评估。该演算还提供了局部近似误差的KL证书、精确的查询条件梯度步表示（其曲率识别出保持注意力的查询方向）以及最小范数查询控制。稀疏评估支持在显式局部失真准则下的候选排序和缓存决策。

英文摘要

We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative $\partial_p z(p) = A z(p)$: its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term $C_{KV} = \sum_j (p'_j - p_j)\,\varepsilon_j$, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.

发表机构

• Hassana Labs（哈萨纳实验室）
• Royal Holloway, University of London（伦敦大学皇家霍洛威学院）
• University of Oxford（牛津大学）

机构由 AI 辅助整理，请以论文原文为准。