CLaM-TTS：改进用于零样本文本转语音的神经编解码器语言模型
CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech
AI总结：
本文提出 CLaM-TTS，通过概率残差向量量化压缩音频 token 并支持多 token 并行生成，以提升零样本 TTS 的可扩展性，在多项指标上达到或超过现有神经编解码器 TTS。
摘要：
随着神经音频编解码器的出现，这类编解码器可从音频中编码多流离散 token，大语言模型近期作为一种用于零样本文本转语音（TTS）合成的有前景方法而受到关注。尽管当前研究纷纷涌向扩展范式，但具有讽刺意味的是，音频 token 化因其较长的序列长度以及对多个序列进行建模的复杂性，反而放大了可扩展性挑战。为缓解这些问题，我们提出 CLaM-TTS，它采用概率残差向量量化，以（1）在 token 长度上实现更优压缩，并（2）允许语言模型一次生成多个 token，从而无需使用级联建模来处理 token 流的数量。我们的实验结果表明，在自然度、可懂度、说话人相似度和推理速度方面，CLaM-TTS 优于或可媲美最先进的基于神经编解码器的 TTS 模型。此外，我们还考察了语言模型的预训练程度及其文本 token 化策略对性能的影响。
英文摘要：
With the emergence of neural audio codecs, which encode multiple streams of discrete tokens from audio, large language models have recently gained attention as a promising approach for zero-shot Text-to-Speech (TTS) synthesis. Despite the ongoing rush towards scaling paradigms, audio tokenization ironically amplifies the scalability challenge, stemming from its long sequence length and the complexity of modelling the multiple sequences. To mitigate these issues, we present CLaM-TTS that employs a probabilistic residual vector quantization to (1) achieve superior compression in the token length, and (2) allow a language model to generate multiple tokens at once, thereby eliminating the need for cascaded modeling to handle the number of token streams. Our experimental results demonstrate that CLaM-TTS is better than or comparable to state-of-the-art neural codec-based TTS models regarding naturalness, intelligibility, speaker similarity, and inference speed. In addition, we examine the impact of the pretraining extent of the language models and their text tokenization strategies on performances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
