AI 中文总结
本文提出通过行和列尺度场（介观层面）分析 Transformer 权重，揭示幅度在通道间的分布，并展示其与功能组织、训练动态及编辑效应的关联，为理解权重结构提供新坐标。
AI 中文摘要
Transformer 权重的汇总统计量掩盖了幅度在功能通道间的分布，而单个权重数量过多，难以直接比较。我们研究介于两者之间的介观层面：行与列尺度场，即权重矩阵在其通道上的以中位数为中心的对数均方根（log-RMS）轮廓，它与全局尺度和一个完全平衡的核心一起，能够精确表示该矩阵。在四个规模的公开 Pythia 检查点以及来自三个初始化家族的受控运行中，平衡揭示了相似的实测核心幅度轮廓。一个混合桥（mixture bridge），其形式在分析前固定，系数通过拟合得到，能够预测在保留运行和受控网格的数据分支上，汇总形状偏离场宽度的程度。索引字段保留了进一步的结构：它们跨共享功能通道的投影对齐，并且查询/键轮廓遵循重新分配的 RoPE 频率，而非固定的矩阵坐标。训练轨迹显示早期字段形成，随后出现依赖于组件的展宽或衰退。将基于通道的分析扩展到 AdamW 的第二矩，揭示其对数空间行和列因子中的相关功能组织。最后，对冻结检查点的编辑将互惠尺度平衡（保持前向计算不变）与相对通道增益区分开来：展平增益会增加分布内损失，同时保持矩阵范数和平衡核心不变。因此，行和列尺度场将汇总幅度统计与通道组织联系起来，并为跟踪和测试训练后的权重结构提供坐标。
英文摘要
Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study the mesoscopic level between them: row and column scale fields, the median-centred log-RMS profiles of a weight matrix over its channels, which together with a global scale and a full balanced core represent the matrix exactly. Across public Pythia checkpoints at four sizes and controlled runs from three initialization families, balancing reveals similar measured core magnitude profiles. A mixture bridge, with its form fixed before the analysis and its coefficients fitted, predicts the pooled-shape departure from field width on held-out runs and data arms of the controlled grid. The indexed fields retain further structure: they align across projections that share a functional channel, and query/key profiles follow reassigned RoPE frequencies rather than fixed matrix coordinates. Training trajectories show early field formation followed by component-dependent broadening or recession. Extending the channel-based analysis to AdamW's second moment reveals related functional organization in its log-space row and column factors. Finally, edits of a frozen checkpoint separate reciprocal scale balance, which preserves the forward computation, from relative channel gain: flattening the gain increases in-distribution loss while preserving matrix norms and the balanced core. Row and column scale fields thus connect pooled magnitude statistics to channel organization and provide coordinates for tracking and testing trained weight structure.
Comments37 pages, 14 figures, 5 tables. Code and data: https://github.com/tiexinding/NPM-Weibull-public (directory scale_field, tag p5-scale-field-v1)
