HALL-E：用于分钟级零样本文本到语音合成的分层神经编解码语言模型
HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis
AI总结：
本文提出MReQ和HALL-E两种后训练方法降低音频帧率，并构建MinutesSpeech基准数据集，实现了单次推理下稳定的分钟级零样本语音合成。
摘要：
近期，基于大语言模型（LLMs）的文本到语音（TTS）模型通过将自然语言文本转换为离散音频 token 序列，受到了广泛关注，这得益于采用残差向量量化（RVQ）的神经音频编解码器（NAC）模型的进步。然而，由于高帧率增加了音频 token 的长度，使得自回归语言模型甚至难以生成一分钟语音的音频 token，长篇语音合成仍然是一个重大挑战。为了应对这一挑战，本文引入了两种新颖的后训练方法：1）多分辨率重量化（MReQ）和 2）HALL-E。MReQ 是一个用于降低预训练 NAC 模型帧率的框架。具体而言，它包含多分辨率残差向量量化（MRVQ）模块，通过师生蒸馏分层重组离散音频 token。HALL-E 是一个基于 LLM 的 TTS 模型，旨在预测 MReQ 的分层 token。具体而言，它采用了使用 MRVQ 子模块的技术，并从预训练的基于 LLM 的 TTS 模型继续训练。此外，为了促进 TTS 研究，我们创建了 MinutesSpeech，这是一个新的基准数据集，包含 40k 小时的过滤语音数据，用于训练和评估从 3 秒到 180 秒的语音合成。在实验中，我们通过将后训练框架应用于 VALL-E 证明了方法的有效性。我们将帧率降至 8 Hz，实现了在单次推理步骤中稳定的分钟级语音合成。音频样本、数据集、代码和预训练模型可在 https://yutonishimura-v2.github.io/HALL-E_DEMO/ 获取。
英文摘要：
Recently, Text-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio codec (NAC) models using residual vector quantization (RVQ). However, long-form speech synthesis remains a significant challenge due to the high frame rate, which increases the length of audio tokens and makes it difficult for autoregressive language models to generate audio tokens for even a minute of speech. To address this challenge, this paper introduces two novel post-training approaches: 1) Multi-Resolution Requantization (MReQ) and 2) HALL-E. MReQ is a framework to reduce the frame rate of pre-trained NAC models. Specifically, it incorporates multi-resolution residual vector quantization (MRVQ) module that hierarchically reorganizes discrete audio tokens through teacher-student distillation. HALL-E is an LLM-based TTS model designed to predict hierarchical tokens of MReQ. Specifically, it incorporates the technique of using MRVQ sub-modules and continues training from a pre-trained LLM-based TTS model. Furthermore, to promote TTS research, we create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered speech data for training and evaluating speech synthesis ranging from 3s up to 180s. In experiments, we demonstrated the effectiveness of our approaches by applying our post-training framework to VALL-E. We achieved the frame rate down to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single inference step. Audio samples, dataset, codes and pre-trained models are available at https://yutonishimura-v2.github.io/HALL-E_DEMO/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
