arXiv 2512.045522026-02-15cs.SDcs.AIeess.AS

RRPO: 为基于LLM的情感TTS的鲁棒奖励策略优化

RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS

Speech Team, Tongyi Lab, Alibaba Group（通义实验室，阿里集团）

Cong Wang, Changfeng Gao, Yang Xiang, Zhihao Du, Keyu An, Han Zhao, Qian Chen, Xiangang Li, Yingming Gao, Ya Li

备注：: Accepted by ICASSP 2026. Copyright 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RRPO通过鲁棒奖励模型和混合正则化方案，提升LLM情感TTS的鲁棒性和自然度，有效对抗奖励黑客攻击。

摘要：

可微分强化学习（RL）框架如DiffRO为可控的文本到语音（TTS）提供了强大的方法，但容易受到奖励黑客攻击，尤其是对于需要细腻控制的情感任务。策略模型可以利用普通的奖励模型（RM）通过生成声音伪影来获得虚假奖励，但会以降质感知质量为代价。为了解决这个问题，我们提出了鲁棒奖励策略优化（RRPO），一种新的框架，采用混合正则化方案。该方案开发了一个鲁棒的RM，其奖励信号更可靠地与人类感知对齐，迫使策略放弃有害的捷径，转而学习真实情感的复杂特征。我们的消融研究证实了我们RM的增强鲁棒性，这体现在其强大的跨语言泛化能力上。主观评估显示，这种鲁棒RM有效缓解了奖励黑客攻击，使情感表达性和自然度在所有基线模型上显著提升。演示页面：https://lrwinr.github.io/RRPO-CosyVoice。

英文摘要：

Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。