arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.25955eess.AS
收藏

SPEAR: 一种用于学习语音和音频表示的统一自监督学习框架

SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

University of Cambridge(剑桥大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学)

Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland

AI总结:

提出SPEAR框架,通过多码本向量量化从语音和音频教师模型中蒸馏互补知识,并采用非对称预训练损失和令牌混合机制,在SUPERB和HEAR基准上取得领先性能。

摘要:

自监督学习(SSL)显著推进了声学表示学习。然而,大多数现有模型针对语音或音频事件理解进行优化,导致这两个领域之间存在持续差距。我们通过SPEAR(语音和音频表示)解决这一差距,这是一个自监督框架,将从专注于语音的SSL教师和通用音频SSL教师中提取的互补知识蒸馏到单个统一模型中。SPEAR对连续的教师表示应用多码本向量量化,以生成捕获语义和声学信息的细粒度离散令牌。为了有效整合这些异构表示,SPEAR通过非对称预训练损失在给定掩码输入的情况下联合预测它们。我们通过一种新颖的令牌混合机制进一步提高了复杂声音场景中的鲁棒性。大量实验表明,SPEAR始终优于现有的统一语音和音频模型。SPEAR在SUPERB基准上建立了新的最先进水平,在15个任务中的12个上超越了WavLM Large,同时在HEAR基准上取得了具有竞争力的性能。这些结果使SPEAR成为通用语音和音频表示学习的多功能基础。代码和预训练模型将发布。

英文摘要:

Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑