We study the geometric structure of layer updates in deep language models. Rather than analyzing what information is encoded in intermediate representations, we ask how representations change from one layer to the next. We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes.
Across multiple architectures, including Transformers and state-space models, we find that the full layer update is almost perfectly aligned with the tokenwise component, while the residual exhibits substantially weaker alignment, larger angular deviation, and significantly lower projection onto the dominant tokenwise subspace. This indicates that the residual is not merely a small correction, but a geometrically distinct component of the transformation.
This geometric separation has functional consequences: approximation error under the restricted tokenwise model is strongly associated with output perturbation, with Spearman correlations often exceeding 0.7 and reaching up to 0.95 in larger models. Together, these results suggest that most layerwise updates behave like structured reparameterizations along a dominant direction, while functionally significant computation is concentrated in a geometrically distinct residual component.
Our framework provides a simple, architecture-agnostic method for probing the geometric and functional structure of layer updates in modern language models.
Deep Neural Networks are generally trained using iterative gradient updates. Magnitudes of gradients are affected by many factors, including choice of activation functions and initialization. More importantly, gradient magnitudes can greatly differ across layers, with some layers receiving much smaller gradients than others. causing some layers to train slower than others and therefore slowing down the overall convergence. We analytically explain this disproportionality. Then we propose to explicitly train all layers at the same speed, by scaling the gradient w.r.t. every trainable tensor to b…
Johannes Kepler University Linz（林茨约翰开普勒大学）；Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究所（IARAI））；University of Oslo（奥斯陆大学）
We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors. It has three types of energy minima (fixed points of the update): (1) global fixed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) fixed points which store a single pattern. The new update rule is equivalent to the attention mechanism used in transformers.…
In this paper, we present a local geometric analysis to interpret how deep feedforward neural networks extract low-dimensional features from high-dimensional data. Our study shows that, in a local geometric region, the optimal weight in one layer of the neural network and the optimal feature generated by the previous layer comprise a low-rank approximation of a matrix that is determined by the Bayes action of this layer. This result holds (i) for analyzing both the output layer and the hidden layers of the neural network, and (ii) for neuron activation functions with non-vanishing gradients. W…
FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学安全、可靠性与跨学科研究中心（SnT））；Carnegie Mellon University（卡内基梅隆大学）；Edith Cowan University（埃迪斯科文大学）；TU Berlin（柏林工业大学）
Recently discovered polyhedral structures of the value function for finite state-action discounted Markov decision processes (MDP) shed light on understanding the success of reinforcement learning. We investigate the value function polytope in greater detail and characterize the polytope boundary using a hyperplane arrangement. We further show that the value space is a union of finitely many cells of the same hyperplane arrangement and relate it to the polytope of the classical linear programming formulation for MDPs. Inspired by these geometric properties, we propose a new algorithm, Geometri…
LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
Center for Cognitive Interaction Technology, Universität Bielefeld, Germany（认知交互技术中心，比勒菲尔德大学，德国）；Bielefeld University（比勒菲尔德大学）；Department of Computer Science, Royal Holloway, University of London, United Kingdom（计算机科学系，伦敦大学皇家霍洛威学院，英国）
We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The feedback weights are also updated with a local rule, the same as the feedforward weights - a weight is updated solely based on the product of activity of the units it connects. With fixed feedback weights as proposed in Lillicrap et. al (2016) performance degrades quickly as the d…
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
Department of Computer Science（计算机科学系）；Department of Government（政府系）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Department of Statistics and Data Sciences（统计与数据科学系）
The performance of convolutional neural networks (CNN) depends heavily on their architectures. Transfer learning performance of a CNN relies quite strongly on selection of its trainable layers. Selecting the most effective update layers for a certain target dataset often requires expert knowledge on CNN architecture which many practitioners do not posses. General users prefer to use an available architecture (e.g. GoogleNet, ResNet, EfficientNet etc.) that is developed by domain experts. With the ever-growing number of layers, it is increasingly becoming quite difficult and cumbersome to handp…
Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers
University of Technology Sydney（悉尼科技大学）；East China Normal University（华东师范大学）；Shanghai University of Electric Power（上海电力大学）；University of Agder（阿格德大学）；Squirrel AI Learning（松鼠AI）
KAUST Center of Excellence for Generative AI（卡斯土尼亚大学生成人工智能卓越中心）；Shenzhen University School of Mathematical Sciences（深圳大学数学科学学院）；King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹大学科学与技术大学）
Multitask learning is a methodology to boost generalization performance and also reduce computational intensity and memory usage. However, learning multiple tasks simultaneously can be more difficult than learning a single task because it can cause imbalance among tasks. To address the imbalance problem, we propose an algorithm to balance between tasks at the gradient level by applying gradient-based meta-learning to multitask learning. The proposed method trains shared layers and task-specific layers separately so that the two layers with different roles in a multitask network can be fitted t…
ARC Training Centre for Information Resilience (CIRES)（信息韧性ARC培训中心（CIRES））；School of Mathematics and Physics, University of Queensland（昆士兰大学数学与物理学院）；Institute for Molecular Bioscience, University of Queensland（昆士兰大学分子生物科学研究所）；Profenso（Profenso公司）；QIMR Berghofer Medical Research Institute（QIMR伯格霍尔医学研究中心）
Purdue University（普渡大学）；University of Southern California（南加州大学）
Secure aggregation promises a heightened level of privacy in federated learning, maintaining that a server only has access to a decrypted aggregate update. Within this setting, linear layer leakage methods are the only data reconstruction attacks able to scale and achieve a high leakage rate regardless of the number of clients or batch size. This is done through increasing the size of an injected fully-connected (FC) layer. However, this results in a resource overhead which grows larger with an increasing number of clients. We show that this resource overhead is caused by an incorrect perspect…
Geometric Neural Operators via Lie Group-Constrained Latent Dynamics
School of Information and Software Engineering, University of Electronic Science and Technology of China（信息与软件工程学院，电子科学与技术大学）；Computer Science and Engineering, University of Electronic Science and Technology of China（计算机科学与工程，电子科学与技术大学）；Institute of Software Chinese Academy of Sciences, Beijing, China（软件研究所，中国科学院）；School of Robotics and Advanced Manufacture, Harbin Institute of Technology, Shenzhen, China（机器人与先进制造学院，哈尔滨工业大学（深圳））；Department of Computer Science, University of Oxford, Oxford, United Kingdom（计算机科学系，牛津大学）
Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates
University of Science and Technology of China（中国科学技术大学）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Hefei National Laboratory（合肥国家实验室）
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）
For Markov chain Monte Carlo methods, one of the greatest discrepancies between theory and system is the scan order - while most theoretical development on the mixing time analysis deals with random updates, real-world systems are implemented with systematic scans. We bridge this gap for models that exhibit a bipartite structure, including, most notably, the Restricted/Deep Boltzmann Machine. The de facto implementation for these models scans variables in a layerwise fashion. We show that the Gibbs sampler with a layerwise alternating scan order has its relaxation time (in terms of epochs) no …
China University of Mining and Technology（中国矿业大学）；University of Surrey（萨里大学）
Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server communication since the end devices typically have very limited communication bandwidth. This paper presents an enhanced federated learning technique by proposing a synchronous learning strategy on the clients and a temporally weighted aggregation of the local models on the server…
Artificial neural networks have successfully tackled a large variety of problems by training extremely deep networks via back-propagation. A direct application of back-propagation to spiking neural networks contains biologically implausible components, like the weight transport problem or separate inference and learning phases. Various methods address different components individually, but a complete solution remains intangible. Here, we take an alternate approach that avoids back-propagation and its associated issues entirely. Recent work in deep learning proposed independently training each …
Microsoft Research India（微软印度研究院）；IIT Kanpur（印度理工学院坎普尔分校）
Training multi-layer Graph Convolution Networks (GCN) using standard SGD techniques scales poorly as each descent step ends up updating node embeddings for a large portion of the graph. Recent attempts to remedy this sub-sample the graph that reduces compute but introduce additional variance and may offer suboptimal performance. This paper develops the IGLU method that caches intermediate computations at various GCN layers thus enabling lazy updates that significantly reduce the compute cost of descent. IGLU introduces bounded bias into the gradients but nevertheless converges to a first-order…
Free University of Berlin（柏林自由大学）；RUDN University（俄罗斯人民友谊大学）
The paper deals with learning probability distributions of observed data by artificial neural networks. We suggest a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We establish a connection between the gradient conjugate prior update and the maximization of the log-likelihood of the predictive distribution. Unlike for the Bayesian neural networks, we use deterministic weights of neural networks, but rather assume that the ground truth distribution is normal with unknown mean and var…
