State Space Model for New-Generation Network Alternative to Transformers: A Survey
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Peng Cheng Laboratory(鹏城实验室);Harbin Institute of Technology (HITSZ)(哈尔滨工业大学(深圳));School of Computer Science, Peking University(北京大学计算机学院);School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University(北京大学深圳研究生院电子与计算机工程学院)
Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, Ziwen Wang, Bo Jiang, Chenglong Li, Yaowei W… 展开作者
Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, Ziwen Wang, Bo Jiang, Chenglong Li, Yaowei Wang, Yonghong Tian, Jin Tang
备注:
The First review of State Space Model (SSM)/Mamba and their applications in artificial intelligence, 33 pages
在后深度学习时代,Transformer架构已在预训练大模型及各类下游任务中展现出强大性能。然而,该架构的巨大计算需求令许多研究者望而却步。为进一步降低注意力模型的复杂度,学界已开展大量工作设计更高效的方法。其中,状态空间模型(State Space Model, SSM)作为基于自注意力的Transformer模型的潜在替代方案,近年来受到越来越多的关注。本文首次对相关工作进行全面综述,并提供实验对比与分析以更好地展现SSM的特性与优势。具体而言,我们首先详细阐述其原理,帮助读者快速掌握SSM的核心思想;随后深入梳理现有SSM及其各类应用,涵盖自然语言处理、计算机视觉、图、多模态与多媒体、点云/事件流、时序数据等多个领域。此外,我们对这些模型进行了统计对比与分析,希望帮助读者理解不同结构在各类任务上的有效性。最后,我们提出了该方向可能的研究点,以更好地推动SSM的理论模型与应用发展。更多相关工作将持续更新在以下GitHub仓库:https://github.com/Event-AHU/Mamba_State_Space_Model_Paper_List。
英文摘要:
In the post-deep learning era, the Transformer architecture has demonstrated its powerful performance across pre-trained big models and various downstream tasks. However, the enormous computational demands of this architecture have deterred many researchers. To further reduce the complexity of attention models, numerous efforts have been made to design more efficient methods. Among them, the State Space Model (SSM), as a possible replacement for the self-attention based Transformer model, has drawn more and more attention in recent years. In this paper, we give the first comprehensive review of these works and also provide experimental comparisons and analysis to better demonstrate the features and advantages of SSM. Specifically, we first give a detailed description of principles to help the readers quickly capture the key ideas of SSM. After that, we dive into the reviews of existing SSMs and their various applications, including natural language processing, computer vision, graph, multi-modal and multi-media, point cloud/event stream, time series data, and other domains. In addition, we give statistical comparisons and analysis of these models and hope it helps the readers to understand the effectiveness of different structures on various tasks. Then, we propose possible research points in this direction to better promote the development of the theoretical model and application of SSM. More related works will be continuously updated on the following GitHub: https://github.com/Event-AHU/Mamba_State_Space_Model_Paper_List.
Alternative positional encoding functions for neural transformers
Department of Computer Languages and Computer Science University of Málaga(计算机语言与计算机科学系马拉加大学);Department of Systems Engineering and Automation University of Málaga(系统工程与自动化系马拉加大学);ITIS Software. Universidad de Málaga(ITIS软件. 马拉加大学)
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
Concordia University(康考迪亚大学);Khalifa University(哈利法大学);Princess Sumaya University for Technology(苏玛雅公主科技大学);University of Alberta(阿尔伯塔大学);Polish Academy of Sciences(波兰科学院);Istinye University(伊斯廷耶大学);King Abdulaziz University(阿卜杜勒阿齐兹国王大学)
Lenovo Research(联想研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Southeast University(东南大学);Lenovo Ltd.(联想有限公司)
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computing and Mathematical Sciences, Birkbeck, University of London(伦敦大学伯克贝克学院计算与数学科学学院)
Department of Electronics and Computer Engineering(电子与计算机工程系);Shenzhen MSU-BIT University(深圳MSU-BIT大学);School of Computer Science and Technology(计算机科学与技术学院)
Large Sequence Models for Sequential Decision-Making: A Survey
Shanghai Jiao Tong University(上海交通大学);Digital Brain Lab(数字大脑实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);the University of Edinburgh(爱丁堡大学);University College London(伦敦大学学院)
MBZ University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Monash University(莫纳什大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);University of Central Florida(中佛罗里达大学)
School of Software, Dalian University of Technology(大连理工大学软件学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院);Institute of Innovation, Science and Sustainability, Federation University Australia(联邦大学澳大利亚创新、科学与可持续性研究所);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Foreign Languages and School of Software Technology, Dalian University of Technology(大连理工大学外语学院和软件技术学院)
From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
United Arab Emirates University(阿联酋大学);Emirates Center for Mobility Research(阿联酋移动性研究中心)