Speech Slytherin：探究Mamba在语音分离、识别与合成中的性能与效率
Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
AI总结：
该研究针对语音分离、识别、合成三类任务提出Mamba及混合模型，对比相近规模Transformer发现，长语音下Mamba效率更优、性能相当或更好，短语音及文语联合建模场景下表现较差，二者优劣取决于具体问题与模型。
摘要：
在将Mamba与Transformer在多项语音相关任务的性能和效率两方面进行对比之前，就断言Mamba是语音领域Transformer的更优替代方案还为时尚早。为得出这一结论，我们针对三类任务提出并评估了三个模型：用于语音分离的Mamba-TasNet、用于语音识别的ConMamba，以及用于语音合成的VALL-M。我们将它们与相近规模的Transformer在性能、内存和速度上进行了对比。我们的Mamba或Mamba-Transformer混合模型展现出与对应Transformer模型（Sepformer、Conformer和VALL-E）相当或更优的性能。对于时长超过某一阈值的语音，它们在内存和速度上比Transformer更高效，该阈值时长与语音token的分辨率呈负相关。用于分离任务的Mamba效率最高，用于识别任务的Mamba效率最低。此外，我们发现对于时长短于该阈值的语音，Mamba的效率并不优于Transformer，且在需要文本与语音联合建模的模型（如双输入的交叉注意力或掩码注意力）中表现更差。因此，我们认为Mamba与Transformer孰优孰劣取决于具体的问题和模型。代码可在https://github.com/xi-j/Mamba-TasNet和https://github.com/xi-j/Mamba-ASR获取。
英文摘要：
It is too early to conclude that Mamba is a better alternative to transformers for speech before comparing Mamba with transformers in terms of both performance and efficiency in multiple speech-related tasks. To reach this conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet for speech separation, ConMamba for speech recognition, and VALL-M for speech synthesis. We compare them with transformers of similar sizes in performance, memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable or higher performance than their transformer counterparts: Sepformer, Conformer, and VALL-E. They are more efficient than transformers in memory and speed for speech longer than a threshold duration, inversely related to the resolution of a speech token. Mamba for separation is the most efficient, and Mamba for recognition is the least. Further, we show that Mamba is not more efficient than transformer for speech shorter than the threshold duration and performs worse in models that require joint modeling of text and speech, such as cross or masked attention of two inputs. Therefore, we argue that the superiority of Mamba or transformer depends on particular problems and models. Code available at https://github.com/xi-j/Mamba-TasNet and https://github.com/xi-j/Mamba-ASR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
