State-space models (SSMs), such as Mamba (Gu & Dao, 2023), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention. Although SSMs exhibit competitive performance, their in-context learning (ICL) capabilities, a remarkable emergent property of modern language models that enables task execution without parameter optimization, remain underexplored compared to Transformers. In this study, we evaluate the ICL performance of SSMs, focusing on Mamba, against Transformer models across various tasks. Our results show that SSMs perform comparably to Transformers in standard regression ICL tasks, while outperforming them in tasks like sparse parity learning. However, SSMs fall short in tasks involving non-standard retrieval functionality. To address these limitations, we introduce a hybrid model, MambaFormer, that combines Mamba with attention blocks, surpassing individual models in tasks where they struggle independently. Our findings suggest that hybrid architectures offer promising avenues for enhancing ICL in language models.
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；National University of Singapore（新加坡国立大学）；Hong Kong Polytechnic University（香港理工大学）；School of Science, Huzhou Normal University（湖州师范学院理学院）
Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；University Health Network（大学网络健康中心）；Peter Munk Cardiac Centre（彼得·芒克心脏中心）；Department of Laboratory Medicine and Pathobiology, University of Toronto（多伦多大学检验医学与病理生物学系）；AI Hub, University Health Network（大学网络健康中心人工智能中心）
Mamba, a State Space Model (SSM), has recently shown competitive performance to Convolutional Neural Networks (CNNs) and Transformers in Natural Language Processing and general sequence modeling. Various attempts have been made to adapt Mamba to Computer Vision tasks, including medical image segmentation (MIS). Vision Mamba (VM)-based networks are particularly attractive due to their ability to achieve global receptive fields, similar to Vision Transformers, while also maintaining linear complexity in the number of tokens. However, the existing VM models still struggle to maintain both spatial…
Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Institute of High Performance Computing, A*STAR（新加坡高性能计算研究院（新加坡科技研究局））；Lehigh University（理海大学）
Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning
KAIST AI（韩国科学技术院人工智能研究所）；University of Tokyo（东京大学）；RIKEN AIP（日本研究机构人工智能计划）；The Institute of Statistical Mathematics（统计数学研究所）；University of Tokyo/RIKEN AIP（东京大学/日本研究机构人工智能计划）
Hybrid Mamba-Transformer Decoder for Error-Correcting Codes
School of Electrical and Computer Engineering（电气与计算机工程学院）；Ben-Gurion University of the Negev（贝内尔-加利利大学内盖夫分校）；Blavatnik School of Computer Science（Blavatnik计算机科学学院）；Tel Aviv University（特拉维夫大学）
School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（电气与计算机工程学院，内盖夫本· Gurion大学）；IAEC；Tel-Aviv University（特拉维夫大学）；IBM Research（IBM研究院）
