测量神经训练动力学中的结构化可预测性：一项跨 regime 研究
Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study
浏览论文内容
中文总结 AI 辅助
本研究以短程可预测性为度量，结合三类探测方法，在 CIFAR 和 Pythia 70M 模型上揭示了深度网络训练动力学中辅助参数与主体参数的可预测性差异，为训练动力学提供了参数分辨率的诊断方式。
中文摘要 AI 辅助
现代深度网络通过长期更新轨迹进行训练，但其时间组织方式仍未像架构、损失函数或优化器那样得到系统表征。我们将短程可预测性作为时间冗余度的度量，研究近期更新在何处、何时以及在何种训练条件下包含关于近期未来参数运动的信息。我们结合三类互补探测族：位移-方向探测、子空间残差探测和基于预测器的探测，采用符合惯例、经零校准的组水平读数，并将其应用于 CIFAR 上的多轮视觉训练以及公开的 Pythia 预训练检查点。在两种 regime 中，归一化参数和偏置等类向量张量（辅助参数）表现出比矩阵类特征变换权重（主体参数）更简单的短程动力学，后者的可预测行为集中在局部、随时间变化的区域中。探测族内部及之间的一致性，以及与独立轨迹诊断的一致性，表明这些测量捕获了轨迹的内在结构，而探测的差异则区分了互补的时间组织形式。受控 CIFAR 对比进一步显示，架构和训练方案会系统性地调节所测结构。Pythia-70M 的案例研究进一步揭示了一系列与角色、深度和尺度相关的事件，包括主体 ESA 降至随机符号一致性水平以下，以及可预测的 qkv 区域在各层之间的出现与重新分布。这些结果将短程可预测性定位为训练动力学的一种回溯性、参数分辨率的诊断工具。
英文摘要
Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.
