arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.16639eess.AScs.CL
收藏

Codec2Vec: 基于神经语音编解码器的自监督语音表示学习

Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs

Department of Computer Science University of Texas at Austin Texas, USA(计算机科学系 休斯顿大学 德州 美国)

Wei-Cheng Tseng, David Harwath

AI总结:

Codec2Vec通过离散音频编解码单元实现自监督语音表示学习,有效提升存储效率和训练速度,同时在SUPERB基准上表现优异。

摘要:

近期神经音频编解码器的发展不仅实现了更优越的音频压缩,还提升了语音合成技术。研究者现在探索其作为通用声学特征提取器在更广泛语音处理任务中的潜力。在此趋势基础上,我们引入Codec2Vec,这是首个仅依赖离散音频编解码单元的语音表示学习框架。该方法提供了多个优势,包括改进的数据存储和传输效率、更快的训练速度和增强的数据隐私。我们探索了基于掩码预测的各种训练目标推导策略,以深入理解该框架的有效性。在SUPERB基准测试中,Codec2Vec在与连续输入模型竞争性能的同时,将存储需求减少高达16.5倍,将训练时间减少2.3倍,展示了其可扩展性和效率。

英文摘要:

Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑