Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer.
MelTok: 2D Tokenization for Single-Codebook Audio Compression
International Digital Economy Academy (IDEA)(国际数字经济学院);Chinese University of Hong Kong(香港中文大学);Shenzhen Graduate School, Peking University(北京大学深圳研究生院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生学院)
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
National Taiwan University(台湾大学);Massachusetts Institute of Technology(麻省理工学院);Microsoft Corporation(微软公司);Meta;The Chinese University of Hong Kong(香港中文大学);Carnegie Mellon University(卡内基梅隆大学)
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
Carnegie Mellon University(卡内基梅隆大学);Amazon GenAI(亚马逊生成人工智能);James Silberrad Brown Center for Artificial Intelligence(詹姆斯·西伯拉德·布朗人工智能中心);University of Bristol(布里斯托大学);Stanford University(斯坦福大学);Northeastern University(东北大学);New York University(纽约大学)
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Center for Computational & Data Sciences, Independent University, Bangladesh(计算与数据科学中心,独立大学,孟加拉国);Amazon GenAI(亚马逊生成人工智能);Qatar Computing Research Institute(卡塔尔计算研究所);University of Virginia(弗吉尼亚大学)