Grokking -- the abrupt transition from memorization to generalization after extended training -- has been linked to the emergence of low-dimensional structure in learning dynamics. Yet neural network parameters inhabit extremely high-dimensional spaces. How can a low-dimensional learning process produce solutions that resist low-dimensional compression?
We investigate this question in multi-task modular arithmetic, training shared-trunk Transformers with separate heads for addition, multiplication, and a quadratic operation modulo 97. Across three model scales (315K--2.2M parameters) and five weight decay settings, we compare three reconstruction methods: per-matrix SVD, joint cross-matrix SVD, and trajectory PCA.
Across all conditions, grokking trajectories are confined to a 2--6 dimensional global subspace, while individual weight matrices remain effectively full-rank. Reconstruction from 3--5 trajectory PCs recovers over 95\% of final accuracy, whereas both per-matrix and joint SVD fail at sub-full rank. Even when static decompositions capture most spectral energy, they destroy task-relevant structure.
These results show that learned algorithms are encoded through dynamically coordinated updates spanning all matrices, rather than localized low-rank components. We term this the holographic encoding principle: grokked solutions are globally low-rank in the space of learning directions but locally full-rank in parameter space, with implications for compression, interpretability, and understanding how neural networks encode computation.
Nonlocal operator learning for fMRI encoding and decoding tasks
Department of Computer Science, Idaho State University(计算机科学系,爱达荷州立大学);Department of Mathematics and Statistics, Idaho State University(数学与统计学系,爱达荷州立大学)
Shape-informed cardiac mechanics surrogates in data-scarce regimes via geometric encoding and generative augmentation
MOX Laboratory, Department of Mathematics, Politecnico di Milano(米兰Politecnico大学数学系MOX实验室);Division 2.2 Process Simulation, Bundesanstalt für Materialforschung und -prüfung (BAM)(材料研究所(BAM)第2.2过程模拟部);Department of Mathematics, University of Trento(特伦托大学数学系)
SCONE: A Practical, Constraint-Aware Plug-in for Latent Encoding in Learned DNA Storage
Department of Computer Science and Engineering(计算机科学与工程系);College of Software(软件学院);Department of Genetics(遗传学系);Stanford School of Medicine(斯坦福医学院);Futurewei Technologies Inc.(未来科技有限公司)
From Uniform to Learned Knots: A Study of Spline-Based Numerical Encodings for Tabular Deep Learning
BASF(巴斯夫);Clausthal University of Technology(克劳斯塔尔工业大学);Amazon Music(亚马逊音乐);Bielefeld School of Business, Hochschule Bielefeld (HSBI)(比勒费尔德商学院,比勒费尔德应用科学大学)
Interpretable vs Learned Encoders for High-Cardinality Fraud Detection
Goizueta Business School Emory University Atlanta, GA, USA(埃默里大学戈伊苏埃塔商学院);Computer Sciences University of California, Berkeley Kirkland, WA(加州大学伯克利分校计算机科学系);Stern School of Business New York University New York, NY, USA(纽约大学斯特恩商学院);School of Data Science University of Pennsylvania Philadelphia, PA, USA(宾夕法尼亚大学数据科学学院);Pratt School of Engineering Duke University Durham, NC, USA(达特茅斯大学普拉特工程学院)
在IEEE-CIS欺诈数据集上比较7种分类编码方法,实体嵌入在AUC-ROC上最优(0.9612),与CatBoost(0.9602)无显著差异,优于其他编码器;CatBoost在AUC-PR上领先(0.822 vs 0.793),无编码器同时主导两个指标。
Expressivity of Parameterized and Data-driven Representations in Quality Diversity Search
Leiden Institute of Advanced Computer Science(莱顿高级计算机科学研究所);Bonn-Rhein-Sieg University of Applied Sciences(波恩-莱茵-锡格应用科学大学);Queen Mary University of London(伦敦玛丽女王大学);SensiLab, Monash University(莫纳什大学SensiLab)