用于文本到语音合成的自回归扩散Transformer
Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
AI总结：
本文提出ARDiT，将音频编码为连续向量并用自回归扩散Transformer生成，结合IKL蒸馏实现单步采样，在零样本文本到语音和语音编辑上达到或超越最先进模型，同时显著降低采样延迟。
摘要：
音频语言模型近来已成为各类音频生成任务中一种颇具前景的方法，其依赖音频分词器将波形编码为离散符号序列。音频分词往往需要在码率与重建精度之间做出必要折中。在处理低码率音频编码时，语言模型只能处理音频中嵌入信息的一个子集，这反过来限制了其生成能力。为规避这些问题，我们提出将音频编码为连续空间 $\mathbb R^d$ 中的向量序列，并使用仅解码器的扩散Transformer（ARDiT）自回归地生成这些序列。我们的研究结果表明，ARDiT在零样本文本到语音任务中表现出色，其性能可与最先进模型相媲美甚至超越它们。高码率连续语音表示能够实现几乎无瑕疵的重建，使我们的模型能够实现近乎完美的语音编辑。我们的实验揭示，在每个自回归步骤中使用积分Kullback-Leibler（IKL）散度进行蒸馏，能显著提升样本的感知质量。同时，它将扩散模型的迭代采样过程压缩为单步。此外，ARDiT可被训练为在一步中预测多个连续向量，从而显著降低采样延迟。令人印象深刻的是，我们的一个模型能够在每个评估步骤生成170毫秒的24 kHz语音，且性能退化极小。音频样本可在 http://ardit-tts.github.io/ 获取。
英文摘要：
Audio language models have recently emerged as a promising approach for various audio generation tasks, relying on audio tokenizers to encode waveforms into sequences of discrete symbols. Audio tokenization often poses a necessary compromise between code bitrate and reconstruction accuracy. When dealing with low-bitrate audio codes, language models are constrained to process only a subset of the information embedded in the audio, which in turn restricts their generative capabilities. To circumvent these issues, we propose encoding audio as vector sequences in continuous space $\mathbb R^d$ and autoregressively generating these sequences using a decoder-only diffusion transformer (ARDiT). Our findings indicate that ARDiT excels in zero-shot text-to-speech and exhibits performance that compares to or even surpasses that of state-of-the-art models. High-bitrate continuous speech representation enables almost flawless reconstruction, allowing our model to achieve nearly perfect speech editing. Our experiments reveal that employing Integral Kullback-Leibler (IKL) divergence for distillation at each autoregressive step significantly boosts the perceived quality of the samples. Simultaneously, it condenses the iterative sampling process of the diffusion model into a single step. Furthermore, ARDiT can be trained to predict several continuous vectors in one step, significantly reducing latency during sampling. Impressively, one of our models can generate $170$ ms of $24$ kHz speech per evaluation step with minimal degradation in performance. Audio samples are available at http://ardit-tts.github.io/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
