归一化自由变换器中初始化下的亚临界信号传播
Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
AI总结：
研究归一化自由变换器中初始化下的信号传播，通过平均偏导数雅可比范数（APJN）分析双向注意力和排列对称输入token配置下的激活统计和APJN递推关系，揭示注意力对深度下APJN渐进行为的影响，表明替换归一化为元素级tanh-like非线性变换的变换器具有亚临界特性。
摘要：
我们通过平均偏导数雅可比范数（APJN），一种衡量跨层梯度放大的度量，研究变换器在初始化时的信号传播。我们通过推导激活统计和APJN在层间的递推关系，将APJN分析扩展到具有双向注意力和排列对称输入token配置的变换器。我们的理论预测了注意力如何修改APJN在大深度下的渐进行为，并与深度视觉变换器中测量的APJN相吻合。残差网络中已知的临界图景也适用于变换器：预归一化架构表现出幂律APJN增长，而用元素级tanh-like非线性替代归一化的变换器表现出伸缩指数APJN增长，表明后者为亚临界。应用于动态tanh（DyT）和动态erf（Derf）变换器时，该理论解释了为何这些架构对初始化和优化选择更敏感，并需要仔细调参以实现稳定训练。
英文摘要：
We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirectional attention and permutation-symmetric input token configurations by deriving recurrence relations for activation statistics and APJNs across layers. Our theory predicts how attention modifies the asymptotic behavior of the APJN at large depth and matches APJNs measured in deep vision transformers. The criticality picture known from residual networks carries over to transformers: the pre-LayerNorm architecture exhibits power-law APJN growth, whereas transformers with LayerNorm replaced by elementwise $\tanh$-like nonlinearities have stretched-exponential APJN growth, indicating that the latter are subcritical. Applied to Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers, the theory explains why these architectures can be more sensitive to initialization and optimization choices and require careful tuning for stable training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
