arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.18649cs.LGcs.AI
收藏

全局低维,局部全维:学习算法的全息编码

Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms

Yongzhong Xu

AI总结:

该研究通过多任务模块算术验证了学习算法的全息编码原则,发现groke了解决方案在学习方向空间中全局低秩,但在参数空间中局部全秩。

摘要:

在扩展训练后,记忆向泛化过渡的现象(grokking)已被与学习动态中低维结构的出现联系起来。然而,神经网络参数却存在于极高的维度空间中。低维学习过程如何产生对低维压缩具有抗性的解决方案?我们通过多任务模块算术进行研究,训练具有单独头的共享主干Transformer,用于加法、乘法和模97的二次运算。在三个模型规模(315K-2.2M参数)和五个权重衰减设置下,我们比较了三种重建方法:每矩阵SVD、联合交叉矩阵SVD和轨迹PCA。在所有条件下,grokking轨迹被限制在2-6维的全局子空间中,而单个权重矩阵仍保持有效全秩。从3-5个轨迹PCA中重建可恢复超过95%的最终准确率,而每矩阵和联合SVD在子全秩方面失败。即使静态分解捕获了大部分谱能量,它们也破坏了任务相关的结构。这些结果表明,学习的算法是通过动态协调更新所有矩阵来编码的,而不是局部低秩组件。我们称其为全息编码原则:groke了解决方案在学习方向空间中是全局低秩的,但在参数空间中是局部全秩的,这对压缩、可解释性和理解神经网络如何编码计算有启示。

英文摘要:

Grokking -- the abrupt transition from memorization to generalization after extended training -- has been linked to the emergence of low-dimensional structure in learning dynamics. Yet neural network parameters inhabit extremely high-dimensional spaces. How can a low-dimensional learning process produce solutions that resist low-dimensional compression? We investigate this question in multi-task modular arithmetic, training shared-trunk Transformers with separate heads for addition, multiplication, and a quadratic operation modulo 97. Across three model scales (315K--2.2M parameters) and five weight decay settings, we compare three reconstruction methods: per-matrix SVD, joint cross-matrix SVD, and trajectory PCA. Across all conditions, grokking trajectories are confined to a 2--6 dimensional global subspace, while individual weight matrices remain effectively full-rank. Reconstruction from 3--5 trajectory PCs recovers over 95\% of final accuracy, whereas both per-matrix and joint SVD fail at sub-full rank. Even when static decompositions capture most spectral energy, they destroy task-relevant structure. These results show that learned algorithms are encoded through dynamically coordinated updates spanning all matrices, rather than localized low-rank components. We term this the holographic encoding principle: grokked solutions are globally low-rank in the space of learning directions but locally full-rank in parameter space, with implications for compression, interpretability, and understanding how neural networks encode computation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑