Mamba-360：作为长序列建模Transformer替代方案的状态空间模型综述：方法、应用与挑战
Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges
AI总结：
本综述全面回顾了状态空间模型（SSMs）作为长序列建模中Transformer替代方案的发展，将其分为门控、结构化和循环三种架构范式，并总结了SSMs在视觉、语言、医疗等多领域的应用及其在多个基准数据集上的性能表现。
摘要：
序列建模是跨多个领域的关键领域，包括自然语言处理（NLP）、语音识别、时间序列预测、音乐生成和生物信息学。循环神经网络（RNN）和长短期记忆网络（LSTM）在历史上一直主导着机器翻译、命名实体识别（NER）等序列建模任务。然而，transformers凭借其卓越的性能，推动了这一范式的转变。尽管如此，transformers仍存在 $O(N^2)$ 的注意力复杂度以及处理归纳偏置方面的挑战。为解决这些问题，人们提出了若干基于谱网络或卷积的变体，并在一系列任务上表现良好，但它们在处理长序列时仍面临困难。在此背景下，状态空间模型（SSMs）已成为序列建模范式的有力替代方案，特别是随着 S4 及其变体的出现，例如 S4nd、Hippo、Hyena、Diagnol State Spaces (DSS)、Gated State Spaces (GSS)、Linear Recurrent Unit (LRU)、Liquid-S4、Mamba 等。在本综述中，我们将基础 SSMs 划分为三种范式：门控架构、结构化架构和循环架构。本综述还重点介绍了 SSMs 在视觉、视频、音频、语音、语言（尤其是长序列建模）、医疗（包括基因组学）、化学（如药物设计）、推荐系统和时间序列分析（包括表格数据）等多个领域的广泛应用。此外，我们汇总了 SSMs 在基准数据集上的性能，如 Long Range Arena (LRA)、WikiText、Glue、Pile、ImageNet、Kinetics-400、sstv2，以及 Breakfast、COIN、LVU 等视频数据集和各种时间序列数据集。Mamba-360 项目的网页可在此网址获取：https://github.com/badripatro/mamba360。
英文摘要：
Sequence modeling is a crucial area across various domains, including Natural Language Processing (NLP), speech recognition, time series forecasting, music generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short Term Memory Networks (LSTMs) have historically dominated sequence modeling tasks like Machine Translation, Named Entity Recognition (NER), etc. However, the advancement of transformers has led to a shift in this paradigm, given their superior performance. Yet, transformers suffer from $O(N^2)$ attention complexity and challenges in handling inductive bias. Several variations have been proposed to address these issues which use spectral networks or convolutions and have performed well on a range of tasks. However, they still have difficulty in dealing with long sequences. State Space Models(SSMs) have emerged as promising alternatives for sequence modeling paradigms in this context, especially with the advent of S4 and its variants, such as S4nd, Hippo, Hyena, Diagnol State Spaces (DSS), Gated State Spaces (GSS), Linear Recurrent Unit (LRU), Liquid-S4, Mamba, etc. In this survey, we categorize the foundational SSMs based on three paradigms namely, Gating architectures, Structural architectures, and Recurrent architectures. This survey also highlights diverse applications of SSMs across domains such as vision, video, audio, speech, language (especially long sequence modeling), medical (including genomics), chemical (like drug design), recommendation systems, and time series analysis, including tabular data. Moreover, we consolidate the performance of SSMs on benchmark datasets like Long Range Arena (LRA), WikiText, Glue, Pile, ImageNet, Kinetics-400, sstv2, as well as video datasets such as Breakfast, COIN, LVU, and various time series datasets. The project page for Mamba-360 work is available on this webpage.\url{https://github.com/badripatro/mamba360}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
