arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.26296eess.AS
收藏

SPG-Codec:探索语义先验在超低比特率神经语音编码中的作用与边界

SPG-Codec: Exploring the Role and Boundaries of Semantic Priors in Ultra-Low-Bitrate Neural Speech Coding

Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tencent(腾讯)

Mingyu Zhao, Zijian Lin, Kun Wei, Zhiyong Wu

AI总结:

本文研究了在超低比特率下,将冻结语义先验(如HuBERT和Whisper)整合到神经语音编码中的作用与限制,发现语义约束在1.5kbps时可降低WER约10%,但6kbps后效果下降,揭示了实际容量边界,并提出了一种基于比特率的调节策略以平衡语义一致性和感知自然度。

摘要:

传统神经语音编解码器在超低比特率下面临严重的可懂度下降问题,瓶颈从声学失真转为语义损失。为解决此问题,本文系统研究了将冻结语义先验(特别是HuBERT和Whisper)整合到神经语音编码中的作用和基本限制。我们引入并验证了新的语义退休现象:在1.5kbps时,语义约束可将词错误率(WER)降低约10%,但超过6kbps后效果迅速下降,表明了实际容量边界。我们进一步揭示了不同先验类型之间的明显权衡:声学丰富先验(HuBERT)更好地保留了语调和音色细节,而高阶语言先验(Whisper)有效抑制了噪声环境中的语音幻觉(减少幻觉率26%)并显著缩小了对未见说话人的一般化差距。基于这些发现,我们提出了一种基于比特率的调节策略,动态调整先验强度以优化语义一致性和感知自然度之间的权衡。广泛实验评估证实,我们的方法在可懂度和噪声鲁棒性方面与现有基线相比具有竞争力,为超低比特率生成语音编解码提供了系统化的路径。

英文摘要:

Conventional neural speech codecs suffer from severe intelligibility degradation at ultra-low bitrates, where the bottleneck transitions from acoustic distortion to semantic loss. To address this issue, this paper conducts a systematic investigation into the role and fundamental limits of integrating frozen semantic priors -- specifically HuBERT and Whisper -- into neural speech coding. We introduce and quantitatively validate a novel Semantic Retirement phenomenon: while semantic constraints reduce the Word Error Rate (WER) by up to ~10% relatively at 1.5 kbps, their benefits rapidly diminish beyond 6 kbps, indicating a practical capacity boundary. We further uncover a clear trade-off between different prior types: acoustic-rich priors (HuBERT) better preserve prosodic and timbral details, whereas high-level linguistic priors (Whisper) effectively suppress phonetic hallucinations in noisy environments (reducing hallucination rates by 26 percent) and substantially narrow the generalization gap for unseen speakers. Building on these findings, we propose a bitrate-aware regulation strategy that dynamically adjusts prior strength to optimize the trade-off between semantic consistency and perceptual naturalness. Extensive experimental evaluations confirm that our approach achieves competitive intelligibility and noise robustness compared to existing baselines, offering a principled pathway toward ultra-low-bitrate generative speech coding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑