SteerSpeech：生成语音中情感控制的激活引导
Steerspeech: Activation Steering For Emotion Control In Generated Speech
浏览论文内容
中文总结 AI 辅助
SteerSpeech通过注入轻量级引导向量实现TTS推理时的连续情感控制，无需训练，且保持说话者身份，目标情感分数提升1.08-7.12倍。
中文摘要 AI 辅助
预训练的文本到语音（TTS）模型能够生成富有表现力的语音，但推理时的可靠情感控制仍然具有挑战性：提示和参考音频提供粗略且不一致的控制，而专门的调节和模型适应则需要昂贵的训练。我们提出了SteerSpeech，一个轻量级的激活引导框架，通过将引导向量注入隐藏激活来控制情感。对于每个目标情感，我们训练一个轻量级的低秩变换，使用多专家目标，该目标鼓励单调的情感控制，同时保留说话者身份和语言内容，约束引导漂移，并保持TTS主干网络冻结。为了通过离散语音令牌进行优化，我们引入了一个两遍生成-重放流水线，使用直通估计器通过采样令牌反向传播专家监督。在推理时，目标情感的引导方向通过其相应的变换进行优化，并注入基础TTS模型。使用Qwen3-TTS对见过的、未见过的和带口音的说话者进行客观和主观评估，显示出更强的连续情感控制，同时说话者和内容退化有限。SteerSpeech在目标情感分数上达到基线1.08倍至7.12倍，对于代表性情感，在主观上，高强度引导下获得78.1%至96.8%的强度偏好和1.43倍至1.46倍的说话者身份保留。
英文摘要
Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.
发表机构
- Netflix（奈飞）
机构由 AI 辅助整理，请以论文原文为准。
