Neural audio codecs (NACs) provide compact latent speech representations in the form of sequences of continuous vectors or discrete tokens. In this work, we investigate how these two types of speech representations compare when used as training targets for supervised speech enhancement. We consider both autoregressive and non-autoregressive speech enhancement models based on the Conformer architecture, as well as a simple baseline where the NAC encoder is simply fine-tuned for speech enhancement. Our experiments reveal three key findings: predicting continuous latent representations consistently outperforms discrete token prediction; autoregressive models achieve higher quality but at the expense of intelligibility and efficiency, making non-autoregressive models more attractive in practice; and adding encoder fine-tuning yields the strongest enhancement metrics overall, though at the cost of degraded codec reconstruction. The code and audio samples are available online.
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
National Taiwan University（台湾大学）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Corporation（微软公司）；Meta；The Chinese University of Hong Kong（香港中文大学）；Carnegie Mellon University（卡内基梅隆大学）
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Carnegie Mellon University（卡内基梅隆大学）；Renmin University of China（中国人民大学）；Nanyang Technological University（南洋理工大学）；Tokyo Metropolitan University（东京都立大学）；Fudan University（复旦大学）；University of Chicago（芝加哥大学）；National Taiwan University（台湾大学）；Massachusetts Institute of Technology（麻省理工学院）
UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
National University of Singapore（新加坡国立大学）；Tongyi Speech Lab（通义语音实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Collaboration AI, Cisco Systems, Inc.（Collaboration AI，Cisco系统公司）；Hebrew University of Jerusalem, Israel（耶路撒冷希伯来大学）；University of Illinois Urbana-Champaign, United States（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University, United States（卡内基梅隆大学）
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding
University of Science and Technology of China（中国科学技术大学）；National Engineering Research Center of Speech and Language Information Processing（国家语言语音信息处理工程研究中心）
CodecFake+: Codec-Based Resynthesized Data as a Proxy for Detecting CodecFake Speech
Graduate Institute of Communication Engineering, National Taiwan University（国家交通大学通信工程研究院）；Department of Computer Science and Information Engineering, National Taiwan University（国家交通大学计算机科学与信息工程系）；Center for Language and Speech Processing at Johns Hopkins University（约翰霍普金斯大学语言与语音处理中心）；Department of Electrical Engineering, National Taiwan University（国家交通大学电子工程系）；Research Center for Information Technology Innovation, Academia Sinica（学术院信息技术创新研究中心）；NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)（国家交通大学人工智能研究中心）
Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency
Shanghai Jiao Tong University（上海交通大学）；MoE Key Lab of Artificial Intelligence（人工智能摩尔实验室）；Jiangsu Key Lab of Language Computing（江苏语言计算重点实验室）；X-LANCE Lab（X-LANCE实验室）；School of Computer Science（计算机科学学院）
Analysis of Speaker Verification Performance Trade-offs with Neural Audio Codec Transmission
Indian Institute of Science Education and Research, Bhopal（印度科学教育与研究学院，博帕尔）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）；Menlo Research（Menlo研究）
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test w…
APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm
National Engineering Research Center of Speech and Language Information Processing（国家语言语音信息处理工程研究中心）；University of Science and Technology of China（中国科学技术大学）
Fewer-token Neural Speech Codec with Time-invariant Codes
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tsinghua University（清华大学）
In this paper, we propose a deep learning (DL)-based parameter enhancement method for a mixed excitation linear prediction (MELP) speech codec in noisy communication environment. Unlike conventional speech enhancement modules that are designed to obtain clean speech signal by removing noise components before speech codec processing, the proposed method directly enhances codec parameters on either the encoder or decoder side. As the proposed method has been implemented by a small network without any additional processes required in conventional enhancement systems, e.g., time-frequency (T-F) an…
Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Information and Communication Engineering, Communication University of China（中国传媒大学信息与通信工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）；Shanghai Polytechnic University（上海第二工业大学）
