arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09869cs.SDeess.AS
收藏

MMM:来自自监督学习模型的多层、多残差、多流离散语音表示

MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model

Carnegie Mellon University(卡内基梅隆大学);Meta AI

Jiatong Shi, Xutai Ma, Hirofumi Inaguma, Anna Sun, Shinji Watanabe

AI总结:

本文提出 MMM,通过对 SSL 模型不同层使用结合 K-means 的迭代残差向量量化提取多流离散语音单元,并在多项语音任务中达到或超过神经编解码器性能。

摘要:

语音离散表示因其对波形的出色压缩率、训练期间的快速收敛以及与其他模态的兼容性,已在多种下游应用中被证明有效。从自监督学习(SSL)模型中提取的离散单元,已成为获取语音离散表示的一种重要方法。然而,尽管离散单元相较于频谱特征已表现出有效性,它们仍落后于连续的 SSL 表示。在本工作中,我们提出 MMM,这是一种从 SSL 中提取多层、多残差、多流离散单元的方法。具体而言,我们针对 SSL 模型中的不同层引入结合 K-means 的迭代残差向量量化,以提取多流语音离散表示。通过在语音识别、语音再合成和文本到语音中的大量实验,我们证明所提出的 MMM 在多种条件下能够超越神经编解码器,或与其性能相当。

英文摘要:

Speech discrete representation has proven effective in various downstream applications due to its superior compression rate of the waveform, fast convergence during training, and compatibility with other modalities. Discrete units extracted from self-supervised learning (SSL) models have emerged as a prominent approach for obtaining speech discrete representation. However, while discrete units have shown effectiveness compared to spectral features, they still lag behind continuous SSL representations. In this work, we propose MMM, a multi-layer multi-residual multi-stream discrete units extraction method from SSL. Specifically, we introduce iterative residual vector quantization with K-means for different layers in an SSL model to extract multi-stream speech discrete representation. Through extensive experiments in speech recognition, speech resynthesis, and text-to-speech, we demonstrate the proposed MMM can surpass or on-par with neural codec's performance under various conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑