Neural audio codecs (NACs) typically encode the short-term energy (gain) and normalized structure (shape) of speech/audio signals jointly within the same latent space. As a result, they are poorly robust to a global variation of the input signal level in the sense that such a variation has a strong influence on the embedding vectors at the output of the encoder and their quantization. This methodology is inherently inefficient, leading to codebook redundancy and suboptimal bitrate-distortion performance. To address these limitations, we propose to introduce shape-gain decomposition, widely used in classical speech and audio coding, into the NAC framework. The principle of the proposed Equalizer methodology, easily applicable to any NAC, is to decompose the input signal---before the NAC encoder---into gain and normalized shape vector on a short-term basis. The shape vector is processed by the NAC, while the gain is quantized with scalar quantization and transmitted separately. The output (decoded) signal is reconstructed from the normalized output of the NAC and the quantized gain. Our experiments conducted on speech signals with four different prominent codecs show that this general methodology enables a substantial gain in bitrate-distortion performance, as well as a massive reduction in quantizer complexity.
Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Information and Communication Engineering, Communication University of China(中国传媒大学信息与通信工程学院);Department of Automation, Tsinghua University(清华大学自动化系);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学北京信息科学与技术国家研究中心);Shanghai Polytechnic University(上海第二工业大学)
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Hong Kong University of Science and Technology(香港科技大学);University of Science and Technology Beijing(北京科技大学);Hong Kong Baptist University(香港浸会大学);Microsoft(微软公司);Chinese University of Hong Kong(香港中文大学)
针对现有音频 LLM 编解码器因压缩设计导致的语义缺陷问题,提出 X-Codec 方法,通过在 RVQ 前后融入语义特征与损失,显著降低语音合成 WER 并扩展至非语音任务,提升音频生成性能。
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所);NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)(国立清华大学人工智能研究中心);ASUS Intelligent Cloud Services(ASUS智能云服务)
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
National Taiwan University(台湾大学);Massachusetts Institute of Technology(麻省理工学院);Microsoft Corporation(微软公司);Meta;The Chinese University of Hong Kong(香港中文大学);Carnegie Mellon University(卡内基梅隆大学)
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Carnegie Mellon University(卡内基梅隆大学);Renmin University of China(中国人民大学);Nanyang Technological University(南洋理工大学);Tokyo Metropolitan University(东京都立大学);Fudan University(复旦大学);University of Chicago(芝加哥大学);National Taiwan University(台湾大学);Massachusetts Institute of Technology(麻省理工学院)
UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
National University of Singapore(新加坡国立大学);Tongyi Speech Lab(通义语音实验室);Shanghai Jiao Tong University(上海交通大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院)
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
Jiutian Artificial Intelligence Research Institute, China Mobile, China State Key Laboratory of Multimedia Information Processing, Peking University, China(九天人工智能研究院,中国移动,国家多媒体信息处理重点实验室,北京大学)
Collaboration AI, Cisco Systems, Inc.(Collaboration AI,Cisco系统公司);Hebrew University of Jerusalem, Israel(耶路撒冷希伯来大学);University of Illinois Urbana-Champaign, United States(伊利诺伊大学厄巴纳-香槟分校);Carnegie Mellon University, United States(卡内基梅隆大学)
APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding
University of Science and Technology of China(中国科学技术大学);National Engineering Research Center of Speech and Language Information Processing(国家语言语音信息处理工程研究中心)
A Streamable Neural Audio Codec with Residual Scalar-Vector Quantization for Real-Time Communication
National Engineering Research Center of Speech and Language Information Processing(国家语音与语言信息处理工程研究中心);University of Science and Technology of China(中国科学技术大学)