In this paper, we propose and investigate the use of neural audio codec language models for the automatic generation of sample-based musical instruments based on text or reference audio prompts. Our approach extends a generative audio framework to condition on pitch across an 88-key spectrum, velocity, and a combined text/audio embedding. We identify maintaining timbral consistency within the generated instruments as a major challenge. To tackle this issue, we introduce three distinct conditioning schemes. We analyze our methods through objective metrics and human listening tests, demonstrating that our approach can produce compelling musical instruments. Specifically, we introduce a new objective metric to evaluate the timbral consistency of the generated instruments and adapt the average Contrastive Language-Audio Pretraining (CLAP) score for the text-to-instrument case, noting that its naive application is unsuitable for assessing this task. Our findings reveal a complex interplay between timbral consistency, the quality of generated samples, and their correspondence to the input prompt.
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to q…
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
National Taiwan University（台湾大学）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Corporation（微软公司）；Meta；The Chinese University of Hong Kong（香港中文大学）；Carnegie Mellon University（卡内基梅隆大学）
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Carnegie Mellon University（卡内基梅隆大学）；Renmin University of China（中国人民大学）；Nanyang Technological University（南洋理工大学）；Tokyo Metropolitan University（东京都立大学）；Fudan University（复旦大学）；University of Chicago（芝加哥大学）；National Taiwan University（台湾大学）；Massachusetts Institute of Technology（麻省理工学院）
UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
National University of Singapore（新加坡国立大学）；Tongyi Speech Lab（通义语音实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Collaboration AI, Cisco Systems, Inc.（Collaboration AI，Cisco系统公司）；Hebrew University of Jerusalem, Israel（耶路撒冷希伯来大学）；University of Illinois Urbana-Champaign, United States（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University, United States（卡内基梅隆大学）
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding
University of Science and Technology of China（中国科学技术大学）；National Engineering Research Center of Speech and Language Information Processing（国家语言语音信息处理工程研究中心）
Indiana University（印第安纳大学）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Conventional audio coding technologies commonly leverage human perception of sound, or psychoacoustics, to reduce the bitrate while preserving the perceptual quality of the decoded audio signals. For neural audio codecs, however, the objective nature of the loss function usually leads to suboptimal sound quality as well as high run-time complexity due to the large model size. In this work, we present a psychoacoustic calibration scheme to re-define the loss functions of neural audio coding systems so that it can decode signals more perceptually similar to the reference, yet with a much lower m…
Analysis of Speaker Verification Performance Trade-offs with Neural Audio Codec Transmission
Indian Institute of Science Education and Research, Bhopal（印度科学教育与研究学院，博帕尔）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）；Menlo Research（Menlo研究）
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Hong Kong University of Science and Technology（香港科技大学）；University of Science and Technology Beijing（北京科技大学）；Hong Kong Baptist University（香港浸会大学）；Microsoft（微软公司）；Chinese University of Hong Kong（香港中文大学）
针对现有音频 LLM 编解码器因压缩设计导致的语义缺陷问题，提出 X-Codec 方法，通过在 RVQ 前后融入语义特征与损失，显著降低语音合成 WER 并扩展至非语音任务，提升音频生成性能。
Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Information and Communication Engineering, Communication University of China（中国传媒大学信息与通信工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）；Shanghai Polytechnic University（上海第二工业大学）
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test w…
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)（国立清华大学人工智能研究中心）；ASUS Intelligent Cloud Services（ASUS智能云服务）
