This paper explores the capability of Mamba, a recently proposed architecture based on state space models (SSMs), as a competitive alternative to Transformer-based models. In the speech domain, well-designed Transformer-based models, such as the Conformer and E-Branchformer, have become the de facto standards. Extensive evaluations have demonstrated the effectiveness of these Transformer-based models across a wide range of speech tasks. In contrast, the evaluation of SSMs has been limited to a few tasks, such as automatic speech recognition (ASR) and speech synthesis. In this paper, we compared Mamba with state-of-the-art Transformer variants for various speech applications, including ASR, text-to-speech, spoken language understanding, and speech summarization. Experimental evaluations revealed that Mamba achieves comparable or better performance than Transformer-based models, and demonstrated its efficiency in long-form speech processing.
Mamba in Speech: Towards an Alternative to Self-Attention
School of Electrical Engineering and Telecommunications, The University of New South Wales (UNSW)(新南威尔士大学电子工程与电信学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong (Shenzhen)(广东大数据计算省级重点实验室,香港中文大学(深圳));Shenzhen Research Institute of Big data, Shenzhen(深圳大数据研究 institute)
PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition
IIIT-DelhiIndia(印度IIIT-德尔hi大学);V.B.S.P.UIndia(印度V.B.S.P.U大学);UPESIndia(印度UPES大学);Independent ResearcherIndia(印度独立研究者);University of TartuEstonia(爱沙尼亚塔尔图大学);Plaksha UniversityIndia(印度Plaksha大学)
Omni-directional attention mechanism based on Mamba for speech separation
School of Cyberspace Science and Technology, Beijing Institute of Technology(空客空间科学与技术学院,北京理工大学);School of Computer Science, Beijing University of Posts and Telecommunications(计算机学院,北京邮电大学);School of Information and Electronics, Beijing Institute of Technology(信息电子学院,北京理工大学)
Exploring TTS without T Using Biologically/Psychologically Motivated Neural Network Modules (ZeroSpeech 2020)
Primate Research Institute, Kyoto University(京都大学灵长类研究所)
在这项研究中,我们在2020年零资源语音挑战赛中报告了我们对无文本的文本转语音(TTS without T)的探索,其中参与者提出了一个端到端的无监督系统,该系统将语音识别和TTS一起学习。我们使用人工神经网络(ANN)的生物/心理动机模块解决了这一挑战,特别关注人类语言的无监督学习作为一个生物/心理问题。该系统首先用回声状态网络(ESN)处理梅尔倒谱系数(MFCC)帧,并在皮层微电路中模拟计算。结果离散化我们原来的变分自编码器(VAE),实现了Dirichlet的贝叶斯聚类广泛接受的计算语言学和认知科学。离散化的信号,然后恢复成声音波形通过神经网络实现的源滤波器模型的语音生产。
Mamba-SEUNet: Mamba UNet for Monaural Speech Enhancement
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of Electronic Information, Sichuan University(四川大学电子信息学院);Huiyan Technology (Tianjin) Co., Ltd(慧言科技(天津)有限公司);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Keyword Mamba: Spoken Keyword Spotting with State Space Models
School of Computer Science and Communication Engineering, Jiangsu University(江苏大学计算机科学与通信工程学院);Jiangsu Engineering Research Center of Big Data Ubiquitous Perception and Intelligent Agricultural Applications(江苏大数据普适感知与智能农业应用工程研究中心);Provincial Key Laboratory of Computational Intelligence and New Technologies in Low-Altitude Digital Agriculture(省级计算智能与低空数字农业新技术重点实验室)
Deep Active Speech Cancellation with Mamba-Masking Network
Blavatnik School of Computer Science Tel Aviv University(塔尔医学院计算机科学学院);School of Electrical and Computer Engineering Ben-Gurion University of the Negev(本·古里安大学电气与计算机工程学院)
TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms
DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2
School of Media Engineering, Communication University of Zhejiang(浙江传媒学院媒体工程学院);School of Broadcast Announcing Arts, Communication University of Zhejiang(浙江传媒学院播音主持艺术学院);Faculty of Humanities and Arts, Macau University of Science and Technology(澳门科技大学人文艺术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Zhejiang Lab(之江实验室);Changjiang Academy of Art and Design, Shantou University(汕头大学长江艺术与设计学院)
CleanUMamba: A Compact Mamba Network for Speech Denoising using Channel Pruning
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology(电气工程、数学和计算机科学学院,代尔夫特理工大学);Leiden Institute of Advanced Computer Science (LIACS), Leiden University(先进计算机科学研究所(LIACS),莱顿大学)
Temporal-Frequency State Space Duality: An Efficient Paradigm for Speech Emotion Recognition
School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);CCAI, Zhejiang University(浙江大学计算机辅助设计与图形学国家重点实验室(注:CCAI为Computer Aided Design and Computer Graphics的缩写,此处按机构关联翻译));Hefei Zhongjuyuan Intelligent Technology Co., Ltd.(合肥中聚源智能科技有限公司)
DiM-Gesture: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 framework
College of Media Engineering, Communication University of Zhejiang(浙江传媒学院媒体工程学院);School of Broadcast Announcing Arts, Communication University of Zhejiang(浙江传媒学院播音与主持艺术学院);Changjiang Academy of Art and Design, Shantou University(汕头大学长江艺术与设计学院)
Exploring Tradeoffs in Models for Low-latency Speech Enhancement
Google Research(谷歌研究院);Google Chrome Audio(谷歌Chrome音频团队)
我们探讨了各种神经网络配置的单通道和双通道频谱掩模为基础的语音增强。我们的最佳模型在CHiME2语音增强任务上的先前最先进的性能上提高了0.4分贝的信号失真比(SDR)。我们研究权衡,如非因果前瞻,计算和参数计数与增强性能,发现零前瞻模型可以实现,平均而言,在0.03 dB SDR的最佳双向模型。此外,我们发现,200毫秒的前瞻性足以实现与我们最好的双向模型相当的性能。
Mamba for Streaming ASR Combined with Unimodal Aggregation
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)