超越嵌入迁移：Grokking迁移与稳定性中的组件角色
Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability
查看机构详情
- Tianjin Medical University（天津医科大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
本研究通过模算术跨算子迁移实验，发现内部注意力/MLP权重迁移能加速泛化，但冻结表示载体或采用在线门控可显著提升稳定性，揭示了迁移加速与轨迹稳定性之间的组件级分离。
中文摘要 AI 辅助
热启动迁移可以使算法任务快速泛化，但尚不清楚哪些模型组件提供了这种增益，以及该增益在持续优化下是否保持稳定。我们研究了模算术上的跨算子迁移，并区分了有效性（早期速度）与稳定性（达到后回撤）。在一项规模匹配的108次运行实验中，跨越12个种子块（96次运行的2^3因子设计加上12次运行的规模对照），将内部注意力/MLP权重（B）与token嵌入和读出层（E+U）一起迁移，将早期准确率提高了5.46个百分点（Holm p=0.0039），并将确认延迟缩短了558步（Holm p=0.0088）。虽然读出层加内部块迁移在1层模型中满足预先指定的±500步延迟等效标准（TOST p=0.0011，尽管Full在11/12个配对种子中更快），一项前瞻性的2层复制实验确认了内部块的优势（12/12个种子，+704.67积分单位，p=4.88x10^-4），同时揭示了一个架构边界：省略供体嵌入比Full低4475.6个单位，超出±250单位的边界。持续的目标训练经常触发严重的grokking后复发。冻结迁移的表示载体（E，U）几乎消除了离线复发（19.40%降至0.07%，Holm p=0.005859）。在线验证触发的门控将2a+b上的真实最大回撤从22.06%降至0.60%（p=0.000488），前瞻性确认将保护扩展到仿射、非线性二次和2层目标（降低10.94-23.47个百分点），区分了持续稳定化与静态早停。在非阿贝尔S_5中，无屏蔽迁移瞬时激增（峰值95.4%），但前瞻性屏蔽队列未产生确认的收益（+0.15 ± 1.14个百分点）。这些结果确立了迁移加速与轨迹稳定性之间的组件级分离，并揭示了参数屏蔽的经验边界。
英文摘要
Warm-start transfer can make algorithmic tasks generalize rapidly, yet it is unclear which model components provide the gain and whether that gain remains stable under continued optimization. We study cross-operator transfer on modular arithmetic and separate efficacy (early velocity) from stability (post-reach drawdown). In a scale-matched 108-run battery across 12 seed blocks (96-run 2^3 factorial plus 12-run scale control), transferring internal attention/MLP weights (B) alongside token embeddings and readout (E+U) improves early accuracy by 5.46 pp (Holm p=0.0039) and cuts confirmation latency by 558 steps (Holm p=0.0088). While readout plus internal-block transfer satisfies the pre-specified +/-500-step latency equivalence criterion in 1-layer models (TOST p=0.0011, though Full is faster in 11/12 paired seeds), a prospective 2-layer replication confirms the internal-block advantage (12/12 seeds, +704.67 integral units, p=4.88x10^-4) while revealing an architectural boundary: omitting donor embeddings falls 4475.6 units below Full, outside the +/-250-unit margin. Continued target training frequently triggers severe post-grokking relapse. Freezing transferred representation carriers (E, U) nearly eliminates offline relapse (19.40% -> 0.07%, Holm p=0.005859). Online validation-triggered gating slashes True Max Drawdown from 22.06% to 0.60% on 2a+b (p=0.000488), with prospective confirmations extending protection across affine, nonlinear quadratic, and 2-layer targets (10.94-23.47 pp reductions), distinguishing continual stabilization from static early stopping. In non-abelian S_5, unshielded transfer surges transiently (95.4% peak), but a prospective shielding cohort yields no confirmed benefit (+0.15 +/- 1.14 pp). These results establish a component-level dissociation between transfer acceleration and trajectory stability, and expose the empirical boundaries of parameter shielding.
