发表机构
Spellbrush（斯佩尔布拉什公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究探究基于编解码器的语音语言模型中，预测器式强化学习与人类感知的对齐条件，采用GRPO与CER区间约束，发现单奖励针对性提升对应指标，8最佳重排序性能接近GRPO，为多奖励语音后训练的奖励选择提供诊断方法。
AI 中文摘要
基于编解码器的文本转语音（TTS）模型使语言模型的后训练可应用于语音生成，但目前尚不清楚学习到的感知预测器何时可作为强化学习奖励使用，且不会丧失与人类听众的对齐性。本研究采用组相对策略优化（GRPO），使用学习到的动漫风格、自然度、喜爱度和唤醒度的奖励来探究该问题。为防止感知奖励因转录漂移被优化，我们引入字符错误率（CER）区间约束，并在相同奖励门控下将策略优化与N最佳重排序进行对比。在单奖励运行中，每种奖励主要提升其自身的目标指标，表明主观预测器并非可互换的质量替代物。多评估者A/B测试进一步显示人类迁移效果不均，而奖励差距分析将平均迁移与轴内校准分离：汇总分析中，带符号的奖励差距可显著预测听众选择，而残差CER差距则不能，但各轴校准仍存在异质性。8最佳重排序是强的人类水平基准，感知表现上未明显差于GRPO，表明GRPO应被视为将奖励选择的行为摊销到策略中，而非在感知上均匀优于重排序。这些结果支持将主观语音奖励分析为预测器-轴-基准三元组，并为多奖励语音后训练前的奖励选择提供实用诊断方法。
英文摘要
Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.
CommentsSubmitted to EMNLP 2026
