The Mamba-based model has demonstrated outstanding performance across tasks in computer vision, natural language processing, and speech processing. However, in the realm of speech processing, the Mamba-based model's performance varies across different tasks. For instance, in tasks such as speech enhancement and spectrum reconstruction, the Mamba model performs well when used independently. However, for tasks like speech recognition, additional modules are required to surpass the performance of attention-based models. We propose the hypothesis that the Mamba-based model excels in "reconstruction" tasks within speech processing. However, for "classification tasks" such as Speech Recognition, additional modules are necessary to accomplish the "reconstruction" step. To validate our hypothesis, we analyze the previous Mamba-based Speech Models from an information theory perspective. Furthermore, we leveraged the properties of HuBERT in our study. We trained a Mamba-based HuBERT model, and the mutual information patterns, along with the model's performance metrics, confirmed our assumptions.
Mamba in Speech: Towards an Alternative to Self-Attention
School of Electrical Engineering and Telecommunications, The University of New South Wales (UNSW)(新南威尔士大学电子工程与电信学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong (Shenzhen)(广东大数据计算省级重点实验室,香港中文大学(深圳));Shenzhen Research Institute of Big data, Shenzhen(深圳大数据研究 institute)
PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition
IIIT-DelhiIndia(印度IIIT-德尔hi大学);V.B.S.P.UIndia(印度V.B.S.P.U大学);UPESIndia(印度UPES大学);Independent ResearcherIndia(印度独立研究者);University of TartuEstonia(爱沙尼亚塔尔图大学);Plaksha UniversityIndia(印度Plaksha大学)
Omni-directional attention mechanism based on Mamba for speech separation
School of Cyberspace Science and Technology, Beijing Institute of Technology(空客空间科学与技术学院,北京理工大学);School of Computer Science, Beijing University of Posts and Telecommunications(计算机学院,北京邮电大学);School of Information and Electronics, Beijing Institute of Technology(信息电子学院,北京理工大学)
Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of Electronic Information, Sichuan University(四川大学电子信息学院);Huiyan Technology (Tianjin) Co., Ltd(慧言科技(天津)有限公司);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative
School of Computing, University of Eastern Finland(计算机学院,东方芬兰大学);Department of Statistics and Applied Probability, University of California Santa Barbara(统计与应用概率系,加州大学圣芭芭拉分校);School of Computer Science and Technology, University of Chinese Academy of Sciences(计算机科学与技术学院,中国科学院大学);Department of Mathematics, University of Toronto(数学系,多伦多大学);Graduate Institute of Communication Engineering, National Taiwan University(通信工程研究生院,台湾大学)
Rethinking Flow and Diffusion Bridge Models for Speech Enhancement
Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University(南京大学声学研究所现代声学重点实验室);NJU-Horizon Intelligent Audio Lab, Horizon Robotics(地平线智能音频实验室);Tencent AI Lab(腾讯AI实验室)
Keyword Mamba: Spoken Keyword Spotting with State Space Models
School of Computer Science and Communication Engineering, Jiangsu University(江苏大学计算机科学与通信工程学院);Jiangsu Engineering Research Center of Big Data Ubiquitous Perception and Intelligent Agricultural Applications(江苏大数据普适感知与智能农业应用工程研究中心);Provincial Key Laboratory of Computational Intelligence and New Technologies in Low-Altitude Digital Agriculture(省级计算智能与低空数字农业新技术重点实验室)
Deep Active Speech Cancellation with Mamba-Masking Network
Blavatnik School of Computer Science Tel Aviv University(塔尔医学院计算机科学学院);School of Electrical and Computer Engineering Ben-Gurion University of the Negev(本·古里安大学电气与计算机工程学院)
Rethinking Speech Representation Aggregation in Speech Enhancement: A Phonetic Mutual Information Perspective
Department of Intelligence and Information, Seoul National University, Republic of Korea(智能与信息系,首尔国立大学,韩国);Artificial Intelligence Institute, Seoul National University, Republic of Korea(人工智能研究所,首尔国立大学,韩国)
DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
School of Media Engineering, Communication University of Zhejiang(浙江传媒学院媒体工程学院);School of Broadcast Announcing Arts, Communication University of Zhejiang(浙江传媒学院播音主持艺术学院);Faculty of Humanities and Arts, Macau University of Science and Technology(澳门科技大学人文艺术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Zhejiang Lab(之江实验室);Changjiang Academy of Art and Design, Shantou University(汕头大学长江艺术与设计学院)
TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms
Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA(语言与语音处理中心(CLSP),约翰霍普金斯大学,美国);Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA(稳健语音系统中心(CRSS),德克萨斯大学达拉斯分校,美国);Language Technologies Institute (LTI), Carnegie Mellon University, USA(语言技术研究所(LTI),卡内基梅隆大学,美国)
Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics
Fraunhofer Institute for Integrated Circuits (IIS)(弗劳恩霍夫集成电路研究所 (IIS));Jade University of Applied Sciences(亚德应用科学大学);Carl von Ossietzky Universität Oldenburg(奥尔登堡卡尔·冯·奥西茨基大学)
CleanUMamba: A Compact Mamba Network for Speech Denoising using Channel Pruning
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology(电气工程、数学和计算机科学学院,代尔夫特理工大学);Leiden Institute of Advanced Computer Science (LIACS), Leiden University(先进计算机科学研究所(LIACS),莱顿大学)