多奖励GRPO用于大规模稳定且有声调的单代码本TTS LLMs
Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
AI总结：
本文提出多奖励GRPO框架，通过优化单代码本TTS LLMs的生成策略，提升语音的语调稳定性、说话者相似性和自然度。
摘要：
近年来，大型语言模型（LLMs）的进步彻底改变了文本到语音（TTS）合成，启发了将语音表示为离散编码器令牌序列的自回归框架。其中，单代码本TTS LLMs已发展为紧凑且可流式传输的架构，共同建模语义和声学整合。然而，尽管这些模型效率高，但往往表现出不稳定的语调、说话者漂移和降质的自然性。为了解决这些问题，我们提出了一种多奖励组相对策略优化（GRPO）框架，直接优化单代码本TTS LLMs的令牌生成策略。除了标准的可理解性和说话者相似性目标外，我们的设计集成了三个基于规则的奖励：持续时间一致性长度惩罚、解码稳定性熵正则化奖励，以及一个LLM标注的语调对齐奖励，该奖励显式监督节奏。在语调奖励中，一个外部推理LLM通过上下文学习预测多个可能的停顿结构，为GRPO训练提供与人类偏好对齐的监督信号。为了评估通用性，我们进一步在GRPO优化的AR主干上附加了一个流匹配（FM）解码器，并观察到一致的额外增益，表明我们的强化优化增强了内在的AR策略。我们进一步在数据大小和模型规模上进行了可扩展性分析，发现所提出的方法在单代码本TTS LLMs中一致增强了语调稳定性、说话者相似性和整体语音自然性。
英文摘要：
Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
