Transformers are the cornerstone of modern large language models, but their quadratic computational complexity limits efficiency in long-sequence processing. Recent advancements in Mamba, a state space model (SSM) with linear complexity, offer promising efficiency gains but suffer from unstable contextual learning and multitask generalization. Some works conduct layer-level hybrid structures that combine Transformer and Mamba layers, aiming to make full use of both advantages. This paper proposes TransMamba, a novel sequence-level hybrid framework that unifies Transformer and Mamba through shared parameter matrices (QKV and CBx), and thus could dynamically switch between attention and SSM mechanisms at different token lengths and layers. We design the Memory Converter to bridge Transformer and Mamba by converting attention outputs into SSM-compatible states, ensuring seamless information flow at TransPoints where the transformation happens. The TransPoint scheduling is also thoroughly explored for balancing effectiveness and efficiency. We conducted extensive experiments demonstrating that TransMamba achieves superior training efficiency and performance compared to single and hybrid baselines, and validated the deeper consistency between Transformer and Mamba paradigms at sequence level, offering a scalable solution for next-generation language modeling. Code and data are available at https://github.com/Yixing-Li/TransMamba
Physics and Data Driven Transformer-Mamba Framework for Flow Field
National University of Defense Technology（国防科技大学）；Nanjing Agricultural University（南京农业大学）；National Key Laboratory of Parallel and Distributed Computing（并行与分布式计算国家重点实验室）；National SuperComputer Center in Tianjin（国家超级计算天津中心）
Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography
Concordia Institute for Information Systems Engineering (CIISE)（康科德信息系统工程研究所）；Concordia University（康科德大学）；Thomas Jefferson University Hospital（托马斯·杰斐逊大学医院）
Mamba, a State Space Model (SSM), has recently shown competitive performance to Convolutional Neural Networks (CNNs) and Transformers in Natural Language Processing and general sequence modeling. Various attempts have been made to adapt Mamba to Computer Vision tasks, including medical image segmentation (MIS). Vision Mamba (VM)-based networks are particularly attractive due to their ability to achieve global receptive fields, similar to Vision Transformers, while also maintaining linear complexity in the number of tokens. However, the existing VM models still struggle to maintain both spatial…
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
The University of Chicago（芝加哥大学）；University of New South Wales（新南威尔士大学）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at UC Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）
Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Institute of High Performance Computing, A*STAR（新加坡高性能计算研究院（新加坡科技研究局））；Lehigh University（理海大学）
Hybrid Mamba-Transformer Decoder for Error-Correcting Codes
School of Electrical and Computer Engineering（电气与计算机工程学院）；Ben-Gurion University of the Negev（贝内尔-加利利大学内盖夫分校）；Blavatnik School of Computer Science（Blavatnik计算机科学学院）；Tel Aviv University（特拉维夫大学）
HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
University of Delaware Newark Delaware USA；Thomas Jefferson University \& Nemours Children's Health Wilmington Delaware USA；University of Delaware；Thomas Jefferson University \& Nemours Children's Health
Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics
University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；Digital Financial Information Technology Co.LTD（数字金融信息技术有限公司）；Northern Arizona University（北亚利桑那大学）
Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting
Beijing University of Posts and Telecommunications（北京邮电大学）；China Telecom Research Institute（中国电信研究院）；China Telecom Corporation Intelligent Cloud Network Operation Center（中国电信智能云网运营中心）
