arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16532eess.AScs.LGcs.MMcs.SDeess.SP
收藏

WavTokenizer:一种用于音频语言建模的高效声学离散编解码器分词器

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);Fundamental AI Research (FAIR), Meta(Meta基础AI研究(FAIR))
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang,… 展开作者

Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Zhou Zhao

AI总结:

本文提出WavTokenizer,一种高效声学离散编解码器分词器,通过极端压缩(24kHz音频每秒仅需40或75个令牌)和增强的VQ空间、上下文窗口、注意力网络及多尺度判别器,实现SOTA重建质量和丰富语义信息,适用于音频语言建模。

摘要:

语言模型已被有效应用于对自然信号(如图像、视频、语音和音频)进行建模。这些模型的一个关键组成部分是编解码器分词器,它将高维自然信号压缩为低维离散令牌。在本文中,我们介绍了WavTokenizer,它在音频领域相比之前的SOTA声学编解码器模型具有几个优势:1)极端压缩。通过压缩量化器的层数和离散编解码器的时间维度,24kHz采样率的一秒音频仅需一个量化器,产生40或75个令牌。2)改进的主观质量。尽管令牌数量减少,WavTokenizer仍实现了最先进的重建质量,具有出色的UTMOS分数,并固有地包含更丰富的语义信息。具体来说,我们通过设计更广泛的VQ空间、扩展的上下文窗口和改进的注意力网络,以及引入强大的多尺度判别器和逆傅里叶变换结构来实现这些结果。我们在语音、音频和音乐领域进行了广泛的重建实验。与最先进模型相比,WavTokenizer在各种客观和主观指标上表现出强劲性能。我们还测试了语义信息、VQ利用率和生成模型的适应性。全面的消融研究证实了WavTokenizer中每个模块的必要性。相关代码、演示和预训练模型可在https://github.com/jishengpeng/WavTokenizer获取。

英文摘要:

Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑