arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.14713cs.SDcs.AIcs.MMeess.AS
收藏

StyleSpeech:面向预训练可控文本到语音的参数高效微调

StyleSpeech: Parameter-efficient Fine Tuning for Pre-trained Controllable Text-to-Speech

University of New South Wales(新南威尔士大学)

Haowei Lou, Helen Paik, Wen Hu, Lina Yao

AI总结:

StyleSpeech通过Style Decorator结构和LoRA实现风格与音素特征的高效学习,并提出LLM-MOS自动评估指标,在多个基准上显著超越现有TTS方法。

摘要:

本文介绍了StyleSpeech,一种新颖的文本到语音(TTS)系统,旨在提升合成语音的自然度和准确性。在现有TTS技术的基础上,StyleSpeech引入了一种独特的Style Decorator结构,使深度学习模型能够同时学习风格和音素特征,并通过低秩适应(LoRA)原理提高适应性和效率。LoRA允许在预训练模型中高效地适应风格特征。此外,我们提出了一种新颖的自动评估指标——LLM引导的平均意见得分(LLM-MOS),该指标利用大型语言模型为自动评估TTS系统性能提供客观且稳健的协议。在基准数据集上的大量测试表明,我们的方法在生成自然、准确且高质量的语音方面显著优于现有的最先进基线方法。这些进展不仅推动了当前TTS系统能力的边界,也促进了TTS系统在更具动态性和专业性的场景中的应用,例如交互式虚拟助手、自适应有声读物以及游戏中的定制语音。语音样本可在https://style-speech.vercel.app获取。

英文摘要:

This paper introduces StyleSpeech, a novel Text-to-Speech~(TTS) system that enhances the naturalness and accuracy of synthesized speech. Building upon existing TTS technologies, StyleSpeech incorporates a unique Style Decorator structure that enables deep learning models to simultaneously learn style and phoneme features, improving adaptability and efficiency through the principles of Lower Rank Adaptation~(LoRA). LoRA allows efficient adaptation of style features in pre-trained models. Additionally, we introduce a novel automatic evaluation metric, the LLM-Guided Mean Opinion Score (LLM-MOS), which employs large language models to offer an objective and robust protocol for automatically assessing TTS system performance. Extensive testing on benchmark datasets shows that our approach markedly outperforms existing state-of-the-art baseline methods in producing natural, accurate, and high-quality speech. These advancements not only pushes the boundaries of current TTS system capabilities, but also facilitate the application of TTS system in more dynamic and specialized, such as interactive virtual assistants, adaptive audiobooks, and customized voice for gaming. Speech samples can be found in https://style-speech.vercel.app

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑