具有时不变编码的少令牌神经语音编解码器
Fewer-token Neural Speech Codec with Time-invariant Codes
AI总结：
针对神经语音编解码器令牌过多制约语言模型TTS发展的问题，提出带时不变编码的TiCodec，通过分离时不变信息减令牌号，结合一致性损失增强全局信息捕捉，实现更少令牌下更好的重建与TTS合成效果。
摘要：
基于语言模型的文本转语音（TTS）模型（如VALL-E）凭借其在零样本场景下出色的上下文学习能力受到关注。神经语音编解码器是这类模型的关键组件，可将语音转换为离散令牌表示。然而，编解码器生成的过长令牌序列可能会对预测精度产生负面影响，制约基于语言模型的TTS模型的发展。为解决该问题，本文提出了一种名为TiCodec的新型时不变编码神经语音编解码器。通过将时不变信息编码并量化为独立编码，TiCodec可减少需要编码的帧级信息量，有效降低语音编码的令牌数量。此外，本文引入了时不变编码一致性损失，以增强单条语音内时不变编码的一致性，促使其捕捉更多全局信息，这对零样本TTS任务有益。实验结果表明，TiCodec不仅能以更少的令牌提升重建语音质量，还能提高TTS模型合成语音的相似度和自然度，并降低词错误率。
英文摘要：
Language model based text-to-speech (TTS) models, like VALL-E, have gained attention for their outstanding in-context learning capability in zero-shot scenarios. Neural speech codec is a critical component of these models, which can convert speech into discrete token representations. However, excessive token sequences from the codec may negatively affect prediction accuracy and restrict the progression of Language model based TTS models. To address this issue, this paper proposes a novel neural speech codec with time-invariant codes named TiCodec. By encoding and quantizing time-invariant information into a separate code, TiCodec can reduce the amount of frame-level information that needs encoding, effectively decreasing the number of tokens as codes of speech. Furthermore, this paper introduces a time-invariant encoding consistency loss to enhance the consistency of time-invariant code within an utterance and force it to capture more global information, which can benefit the zero-shot TTS task. Experimental results demonstrate that TiCodec can not only enhance the quality of reconstruction speech with fewer tokens but also increase the similarity and naturalness, as well as reduce the word error rate of the synthesized speech by the TTS model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
