发表机构
Faculty of Business and Economics, The University of Hong Kong（香港大学经管学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究预层归一化循环变压器中层归一化作用，指出其为隐式增益控制，分析稳定性预算及进位约束，通过六个任务实验表明线性进位非深度记忆机制，刻画该说法边界，结果经解析得出并在CPU规模实现中验证。
AI 中文摘要
在预层归一化循环变压器中，循环块内的层归一化充当隐式增益控制器：通过将块的局部利普希茨常数与激活尺度成反比耦合，使递归雅可比矩阵非正规——即使在其算子范数超过1的每个已验证固定点处渐近收缩——因此真正的稳定性预算是谱裕度，而不是算子范数界。随着进位ρ→1，该裕度会耗尽，少数初始化根本不会收敛到固定点，因此对角进位约束ρ(Ā)<1对于完整递归的收敛是必要但不充分的。在包括受控消融在内的六个任务上的训练实验表明，线性进位不是深度记忆机制：梯度下降通过块中更具表现力的非线性递归传递记忆，而使受稳定性约束的进位处于静止状态——进位的作用是稳定，而不是记忆作用。我们刻画了这一说法的边界：在具有逐通道结构的任务上，梯度下降确实会利用进位。所有结果都是通过解析得出的，并在从零开始的CPU规模实现中得到验证；需要在更大规模上进行验证。
英文摘要
In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry $ρ\to 1$, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint $ρ(\bar{A}) < 1$ is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.
Comments23 pages, 6 figures, 13 tables. Code available at https://github.com/clojure-finance/mythjure/tree/paper-v1
