CodecSep: 基于提示的通用神经音频编解码器潜在空间声音分离
CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
AI总结：
CodecSep通过在神经音频编解码器潜在空间中直接提取声源，实现开放词汇声音分离，相比AudioSep在SI-SDR指标上表现更优，且在ViSQOL和MOS-LQS上取得显著提升，同时提供低延迟的代码流部署方案。
摘要：
文本引导的声音分离能够实现灵活的音频编辑、辅助听觉和开放领域源提取，但如AudioSep等系统在低延迟边缘或编解码器介质部署中过于昂贵。现有的神经音频编解码器分离器虽然高效，但大多局限于固定声源或封闭分类。我们引入CodecSep，一种基于提示的通用声音分离框架，能够在神经音频编解码器潜在空间中直接提取声源。CodecSep结合了一个冻结的DAC主干网络和一个轻量级的FiLM条件化Transformer掩码器，该掩码器由CLAP文本嵌入驱动，从而实现开放词汇分离，同时保持编解码器原生的效率。在dnr-v2和五个开放领域基准测试中，CodecSep在SI-SDR指标上持续优于AudioSep，在ViSQOL指标上保持竞争力，并在人类MOS-LQS指标上取得明显优势。受控分析显示，细粒度提示优于粗粒度标签，且显式潜在空间掩码比解码器风格的潜在空间生成在编解码器空间中更具有效性。定性诊断显示，神经音频编解码器潜在空间保留了源依赖的结构，CodecSep主要通过通道级源条件化调制来利用该结构。CodecSep还提供了一种实用的代码流部署路径。当音频以神经音频编解码器代码形式传输时，CodecSep将代码映射到嵌入，直接在编解码器空间中分离，并输出波形或重新量化代码，避免了解码-分离-重编码循环。在此模式下，CodecSep仅需1.35 GMACs的端到端计算：与AudioSep在相同管道中的计算量相比减少了约54倍，单独分离器的计算量降低了25倍，同时具有更低的延迟和内存需求。更广泛地说，CodecSep为编解码器原生的下游音频处理提供了蓝图。
英文摘要：
Text-guided sound separation enables flexible audio editing, assistive listening, and open-domain source extraction, but systems such as AudioSep remain too expensive for low-latency edge or codec-mediated deployment. Existing neural audio codec separators are efficient, yet largely restricted to fixed stems or closed taxonomies. We introduce CodecSep, a prompt-driven universal sound separation framework that extracts sources directly in neural audio codec latent space. CodecSep combines a frozen DAC backbone with a lightweight FiLM-conditioned Transformer masker driven by CLAP text embeddings, enabling open-vocabulary separation while preserving codec-native efficiency. Across dnr-v2 and five open-domain benchmarks, CodecSep consistently improves over AudioSep in SI-SDR, remains competitive in ViSQOL, and achieves clear gains in human MOS-LQS. Controlled analyses show that fine-grained prompts outperform coarse labels, and that explicit latent masking is substantially more effective than decoder-style latent generation in codec space. Qualitative diagnostics show that neural audio codec latents retain source-dependent structure, which CodecSep exploits mainly through channel-wise source-conditioned modulation. CodecSep also provides a practical code-stream deployment path. When audio is transmitted as neural audio codec codes, CodecSep maps codes to embeddings, separates directly in codec space, and outputs waveforms or re-quantized codes, avoiding the decode-separate-re-encode loop. In this regime, CodecSep requires only 1.35 GMACs end-to-end: about 54 times less compute than AudioSep in the same pipeline and 25 times lower separator-only compute, with much lower latency and memory. More broadly, CodecSep offers a blueprint for codec-native downstream audio processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
