arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.07273eess.AS
收藏

通过自监督模型重新思考语音处理中的Mamba

Rethinking Mamba in Speech Processing by Self-Supervised Models

Xiangyu Zhang, Jianbo Ma, Mostafa Shahin, Beena Ahmed, Julien Epps

AI总结:

该研究针对语音处理中Mamba模型任务表现差异的问题,提出其擅长重建类任务、分类任务需额外重建模块的假设,通过信息论分析与基于Mamba的HuBERT模型实验验证了假设。

摘要:

基于Mamba的模型已在计算机视觉、自然语言处理和语音处理任务中展现出优异性能。但在语音处理领域,基于Mamba的模型在不同任务上的表现存在差异:例如在语音增强、频谱重建等任务中,Mamba模型独立使用时表现良好;但对于语音识别等任务,需要额外模块才能超越基于注意力的模型的性能。我们提出假设:基于Mamba的模型擅长语音处理中的“重建”类任务,而对于语音识别这类“分类任务”,需要额外模块来完成“重建”步骤。为验证该假设,我们从信息论角度分析了已有的基于Mamba的语音模型,还在研究中利用了HuBERT的特性,训练了一个基于Mamba的HuBERT模型,其互信息模式与模型性能指标均证实了我们的假设。

英文摘要:

The Mamba-based model has demonstrated outstanding performance across tasks in computer vision, natural language processing, and speech processing. However, in the realm of speech processing, the Mamba-based model's performance varies across different tasks. For instance, in tasks such as speech enhancement and spectrum reconstruction, the Mamba model performs well when used independently. However, for tasks like speech recognition, additional modules are required to surpass the performance of attention-based models. We propose the hypothesis that the Mamba-based model excels in "reconstruction" tasks within speech processing. However, for "classification tasks" such as Speech Recognition, additional modules are necessary to accomplish the "reconstruction" step. To validate our hypothesis, we analyze the previous Mamba-based Speech Models from an information theory perspective. Furthermore, we leveraged the properties of HuBERT in our study. We trained a Mamba-based HuBERT model, and the mutual information patterns, along with the model's performance metrics, confirmed our assumptions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑