arXiv 2404.032042024-05-19eess.AScs.AIcs.CLcs.LGcs.SD

RALL-E：面向语音合成的基于思维链提示的鲁棒编解码语言建模

RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis

Microsoft（微软公司）；The University of Tokyo（东京大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong（香港中文大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））

Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, Sheng Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大语言模型驱动的零样本TTS鲁棒性差的问题，提出RALL-E方法，通过思维链提示拆解任务并结合时长引导的自注意力，显著降低词错率与难句错误率。

摘要：

我们提出了RALL-E，一种用于文本转语音（TTS）合成的鲁棒语言建模方法。此前基于大语言模型（LLM）的工作在零样本TTS上展现了出色性能，但这类方法因语言模型的自回归预测模式，往往存在鲁棒性差的问题，比如韵律不稳定（音高、节奏/时长异常）以及词错率（WER）较高。RALL-E的核心思路是思维链（CoT）提示，它将任务拆解为更简单的步骤，以提升基于LLM的TTS的鲁棒性。为实现这一思路，RALL-E首先预测输入文本的韵律特征（音高和时长），并将其作为中间条件，以CoT的方式预测语音token。其次，RALL-E利用预测得到的时长提示来引导Transformer中自注意力权重的计算，促使模型在预测语音token时聚焦于对应的音素和韵律特征。全面的客观与主观评估结果表明，与性能强劲的基线方法VALL-E相比，RALL-E将零样本TTS的WER分别从5.6%（无重排序）和1.7%（有重排序）显著降至2.5%和1.0%。此外，我们证明RALL-E能正确合成VALL-E难以处理的句子，将错误率从68%降至4%。

英文摘要：

We present RALL-E, a robust language modeling method for text-to-speech (TTS) synthesis. While previous work based on large language models (LLMs) shows impressive performance on zero-shot TTS, such methods often suffer from poor robustness, such as unstable prosody (weird pitch and rhythm/duration) and a high word error rate (WER), due to the autoregressive prediction style of language models. The core idea behind RALL-E is chain-of-thought (CoT) prompting, which decomposes the task into simpler steps to enhance the robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts prosody features (pitch and duration) of the input text and uses them as intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E utilizes the predicted duration prompt to guide the computing of self-attention weights in Transformer to enforce the model to focus on the corresponding phonemes and prosody features when predicting speech tokens. Results of comprehensive objective and subjective evaluations demonstrate that, compared to a powerful baseline method VALL-E, RALL-E significantly improves the WER of zero-shot TTS from $5.6\%$ (without reranking) and $1.7\%$ (with reranking) to $2.5\%$ and $1.0\%$, respectively. Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from $68\%$ to $4\%$.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。