LiveSpeech：基于音频离散码自回归建模的低延迟零样本文本转语音
LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes
AI总结：
LiveSpeech提出一种基于全自回归语言模型的零样本文本转语音方法，通过自适应码本损失权重和码本分组并行处理实现低延迟流式输出，在内容准确性、说话人相似度、音频质量和推理速度上达到先进水平。
摘要：
先前的工作已经通过使用基于神经音频编解码器获得的音频令牌上的生成式语言模型，展示了零样本文本转语音的能力。然而，将它们适配到低延迟场景仍然具有挑战性。在本文中，我们提出了 LiveSpeech——一种完全基于自回归语言模型的方法，用于零样本文本转语音，能够实现输出音频的低延迟流式传输。为了在单个解码步骤内允许多个令牌预测，我们提出：（1）使用自适应码本损失权重，该权重考虑每一帧中的码本贡献并聚焦于困难实例；（2）对码本进行分组并并行处理各组。实验表明，我们提出的模型在内容准确性、说话人相似度、音频质量和推理速度方面取得了与最先进基线相竞争的结果，同时适用于低延迟流式应用。
英文摘要：
Prior works have demonstrated zero-shot text-to-speech by using a generative language model on audio tokens obtained via a neural audio codec. It is still challenging, however, to adapt them to low-latency scenarios. In this paper, we present LiveSpeech - a fully autoregressive language model-based approach for zero-shot text-to-speech, enabling low-latency streaming of the output audio. To allow multiple token prediction within a single decoding step, we propose (1) using adaptive codebook loss weights that consider codebook contribution in each frame and focus on hard instances, and (2) grouping codebooks and processing groups in parallel. Experiments show our proposed models achieve competitive results to state-of-the-art baselines in terms of content accuracy, speaker similarity, audio quality, and inference speed while being suitable for low-latency streaming applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
