AI 中文总结
该研究扩展了权重空间消融下的跨层交互理论，推导了注意力子块的闭式雅可比边界，在Qwen2.5-1.5B-Instruct上验证，并发现间接对象识别涌现回路，测试了其交互特性。
AI 中文摘要
一篇姊妹论文研究了在理想模型中激活修补与权重空间消融一致的情形，该模型中条件计算通过残差流以加性方式执行。对于该模型中两个载体在结构上存在依赖的一种组合，即注意力头与其所在层的归一化-MLP组合，它推导了精确的一阶交互公式：仅消融MLP时该值为零，同时消融注意力头时该值为二阶有界。该结果仅局限于单个残差块，且仅在小Transformer的合成任务上进行了验证。本文将该结果扩展至突破上述两个限制。首先，对跨多个层的载体进行消融时产生的交互可精确分解为同层块项（每个涉及的层对应一项），以及一个未声称其量级很小的跨层余项。其次，我们针对两层的情形精确分离该余项，将其表示为混合二阶导数的二重积分，并命名了对其进行边界所需的缺失要素：注意力子块的雅可比边界。我们以闭式形式推导了该边界，并针对Qwen2.5-1.5B-Instruct的真实权重进行了验证，未出现任何违反情况，不过我们尚未将其跨层串联。我们还以闭式形式给出了姊妹论文的边界未展示的曲率常数。第三，在同一模型上，我们使用该任务的原始激活修补方法，搜索并发现了一个未被设计入模型的间接对象识别涌现回路，并对其进行了崩溃、解离和交互测试。结果喜忧参半：在所有五个测试实例中出现了一个共享载体，崩溃和解离在大部分但非全部实例中成立，且在五个实例中的三个实例中可测量到非零交互，且该交互发生在姊妹论文定理所覆盖的同层块之外的层对上。
英文摘要
A companion paper studies when activation patching and weight-space ablation agree, inside an idealized model where a conditional computation is carried additively through a residual stream. For the one composition in that model where two carriers are architecturally dependent, an attention head and its own layer's normalization-MLP composition, it derives an exact first-order interaction formula, zero when only the MLP is ablated and second-order bounded when the head is also ablated. That result is confined to a single residual block and checked only on small transformers on a synthetic task. This paper extends the result past both limits. First, the interaction from ablating carriers spanning several layers decomposes exactly into same-block terms, one per touched layer, plus a cross-layer remainder on which the decomposition makes no claim of smallness. Second, we isolate that remainder exactly, for two layers, as a double integral of a mixed second derivative, and name the missing ingredient needed to bound it: a Jacobian bound for the attention sub-block. We derive this bound in closed form and verify it, without a single violation, against Qwen2.5-1.5B-Instruct's real weights, though we do not yet chain it across layers. We also give, in closed form, the curvature constant the companion paper's bound leaves unexhibited. Third, on that same model, we search for and find an emergent circuit for indirect object identification, never designed into it, using the original activation-patching method for this task, and test collapse, dissociation, and interaction on it. The result is mixed: a shared carrier emerges across all five tested instances, collapse and dissociation hold on most but not all, and a nonzero interaction is measurable on three of five, at layer pairs outside the same-block case the companion theorem covers.
Comments18 pages, 2 figures. Part II of a two-part series; see the companion paper "A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations" (Part I)
