Theoretical Foundations of Deep Selective State-Space Models
Imperial College London(伦敦帝国学院);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(ELLIS图宾根研究所);University of Oxford(牛津大学)
Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, Terry Lyons
Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants.
Exhaustive Circuit Mapping of a Single-Cell Foundation Model Reveals Massive Redundancy, Heavy-Tailed Hub Architecture, and Layer-Dependent Differentiation Control
Department of Computer Science, University of Tübingen(图宾根大学计算机科学系)
Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness
Research Ireland – Centre for Research Training in AI (CRT-AI)(爱尔兰研究机构——人工智能研究培训中心(CRT-AI));J.E. Cairnes School of Business & Economics(J.E. Cairnes 商学院);School of Computer Science(计算机科学学院);University of Galway, Ireland(爱尔兰Galway大学)