Conventional neural speech codecs suffer from severe intelligibility degradation at ultra-low bitrates, where the bottleneck transitions from acoustic distortion to semantic loss. To address this issue, this paper conducts a systematic investigation into the role and fundamental limits of integrating frozen semantic priors -- specifically HuBERT and Whisper -- into neural speech coding. We introduce and quantitatively validate a novel Semantic Retirement phenomenon: while semantic constraints reduce the Word Error Rate (WER) by up to ~10% relatively at 1.5 kbps, their benefits rapidly diminish beyond 6 kbps, indicating a practical capacity boundary. We further uncover a clear trade-off between different prior types: acoustic-rich priors (HuBERT) better preserve prosodic and timbral details, whereas high-level linguistic priors (Whisper) effectively suppress phonetic hallucinations in noisy environments (reducing hallucination rates by 26 percent) and substantially narrow the generalization gap for unseen speakers. Building on these findings, we propose a bitrate-aware regulation strategy that dynamically adjusts prior strength to optimize the trade-off between semantic consistency and perceptual naturalness. Extensive experimental evaluations confirm that our approach achieves competitive intelligibility and noise robustness compared to existing baselines, offering a principled pathway toward ultra-low-bitrate generative speech coding.
CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction
National Engineering Research Center of Speech and Language Information Processing(语音和语言信息处理国家工程研究中心);University of Science and Technology of China(中国科学技术大学)
CFMDCTCodec: A Low-Bitrate Neural Speech Codec with Noise-Prior-aware Conditional Flow Matching for MDCT-Spectral Enhancement
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(中国科学技术大学语音及语言信息处理国家工程研究中心);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China(清华大学深圳国际研究生院);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Hong Kong SAR, China(香港中文大学)
Analysis of Speaker Verification Performance Trade-offs with Neural Audio Codec Transmission
Indian Institute of Science Education and Research, Bhopal(印度科学教育与研究学院,博帕尔);Nanyang Technological University, Singapore(南洋理工大学,新加坡);Menlo Research(Menlo研究)
LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec
Shanghai Jiao Tong University(上海交通大学);X-LANCE Lab, MoE Kay Lab of Artificial Intelligence, School of Computer Science(X-LANCE实验室、MoE Kay人工智能实验室、计算机科学学院)
Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,中国传媒大学);Microsoft Research Asia(微软亚洲研究院);State Key Laboratory of Media Convergence and Communication, Communication University of China(媒体融合与通信国家重点实验室,中国传媒大学)
Collaboration AI, Cisco Systems, Inc.(Collaboration AI,Cisco系统公司);Hebrew University of Jerusalem, Israel(耶路撒冷希伯来大学);University of Illinois Urbana-Champaign, United States(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University, United States(卡内基梅隆大学)
Ultra-Low-Bitrate Mel-Spectrogram-based Neural Speech Coding with Flow-Matching-based Refinement and Vocoding-driven Reconstruction
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(语音和语言信息处理国家级工程研究中心,中国科学技术大学)
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding
University of Science and Technology of China(中国科学技术大学);National Engineering Research Center of Speech and Language Information Processing(国家语言语音信息处理工程研究中心)
APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm
National Engineering Research Center of Speech and Language Information Processing(国家语言语音信息处理工程研究中心);University of Science and Technology of China(中国科学技术大学)
Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate
MoE Key Lab of Artificial Intelligence(人工智能MOE重点实验室);Jiangsu Key Lab of Language Computing(江苏语言计算重点实验室);X-LANCE Lab(X-LANCE实验室);School of Computer Science(计算机科学学院);Department of Data Science and Artificial Intelligence(数据科学与人工智能系)