Koel-TTS：通过偏好对齐和无分类器引导增强基于 LLM 的语音生成
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
AI总结：
本文提出 Koel-TTS，利用 ASR 与说话人验证引导的偏好对齐及无分类器引导，增强文本到声学 token 的可控语音合成，并在小数据集训练下超越现有 TTS。
摘要：
尽管自回归语音 token 生成模型能够生成具有显著多样性和自然度的语音，但其固有的可控性不足往往导致幻觉以及不符合条件输入的非期望发声等问题。我们提出 Koel-TTS，这是一套增强型编码器-解码器 Transformer TTS 模型，通过结合由自动语音识别和说话人验证模型引导的偏好对齐技术来应对这些挑战。此外，我们引入无分类器引导，以进一步提升合成结果对文本和参考说话人音频的遵循程度。实验表明，这些优化显著提高了合成语音的目标说话人相似度、可懂度和自然度。值得注意的是，Koel-TTS 直接将文本和上下文音频映射为声学 token；尽管训练所用数据集明显更小，但在上述指标上仍优于最先进的 TTS 模型。音频样本和演示可在我们的网站获取。
英文摘要：
While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as hallucinations and undesired vocalizations that do not conform to conditioning inputs. We introduce Koel-TTS, a suite of enhanced encoder-decoder Transformer TTS models that address these challenges by incorporating preference alignment techniques guided by automatic speech recognition and speaker verification models. Additionally, we incorporate classifier-free guidance to further improve synthesis adherence to the transcript and reference speaker audio. Our experiments demonstrate that these optimizations significantly enhance target speaker similarity, intelligibility, and naturalness of synthesized speech. Notably, Koel-TTS directly maps text and context audio to acoustic tokens, and on the aforementioned metrics, outperforms state-of-the-art TTS models, despite being trained on a significantly smaller dataset. Audio samples and demos are available on our website.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
