DeCodec：将音频编解码器重新构想为通用解耦表示学习器
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
AI总结：
DeCodec通过正交子空间投影和表示交换训练，将音频编解码器转化为通用解耦表示学习器，实现语音与背景声音及语义与副语言的分层解耦，支持灵活特征选择，在语音增强、声音转换、ASR和TTS中展现新能力。
摘要：
通用音频编解码器学习跨音频类型的纠缠表示，而某些特定编解码器提供解耦表示但仅限于语音。然而，现实世界的音频通常包含混合的语音和背景声音，下游任务需要选择性访问这些组件。因此，我们将音频编解码器重新构想为通用解耦表示学习器，以实现在不同音频任务中的可控特征选择。为此，我们引入了DeCodec，一种新颖的神经编解码器，它学习将音频表示解耦为专用于语音和背景声音的正交子空间，并且在语音内部，表示进一步分解为语义和副语言成分。这种层次化解耦允许灵活的特征选择，使DeCodec成为多种音频应用的通用前端。在技术上，基于编解码器框架，DeCodec包含两个关键创新：一个子空间正交投影模块，将输入分解为两个解耦的正交子空间；以及一个表示交换训练过程，确保这两个子空间分别与语音和背景声音相关。这些允许并行RVQ独立量化语音和背景声音成分。此外，我们对语音RVQ采用语义引导，以实现语义和副语言分解。实验结果表明，DeCodec在保持先进信号重建的同时，实现了新能力：通过表示重组实现优越的语音增强和有效的噪声语音一次性声音转换，通过干净语义特征提高ASR鲁棒性，以及在TTS中可控的背景声音保留/抑制。演示页面：https://luo404.github.io/DeCodecV2/
英文摘要：
Universal audio codecs learn entangled representations across audio types, whereas some specific codecs offer decoupled representations but are limited to speech. Real-world audio, however, often contains mixed speech and background sounds, and downstream tasks require selective access to these components. Therefore, we rethink the audio codec as a universal disentangled representation learner to enable controllable feature selection across different audio tasks. To this end, we introduce DeCodec, a novel neural codec that learns to decouple audio representations into orthogonal subspaces dedicated to speech and background sound, and within speech, representations are further decomposed into semantic and paralinguistic components. This hierarchical disentanglement allows flexible feature selection, making DeCodec a universal front-end for multiple audio applications. Technically, built upon a codec framework, DeCodec incorporates two key innovations: a subspace orthogonal projection module that factorizes the input into two decoupled orthogonal subspaces, and a representation swap training procedure that ensures these two subspaces are correlate to the speech and background sound, respectively. These allows parallel RVQs to quantize speech and background sound components independently. Furthermore, we employ semantic guidance to the speech RVQ to achieve semantic and paralinguistic decomposition. Experimental results show that DeCodec maintains advanced signal reconstruction while enabling new capabilities: superior speech enhancement and effective one-shot voice conversion on noisy speech via representation recombination, improved ASR robustness through clean semantic features, and controllable background sound preservation/suppression in TTS. Demo Page: https://luo404.github.io/DeCodecV2/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
