FleSpeech：基于多种提示的灵活可控语音生成
FleSpeech: Flexibly Controllable Speech Generation with Various Prompts
AI总结：
本文提出FleSpeech多阶段语音生成框架，通过多模态提示编码器统一文本、音频和视觉提示，实现对语音属性的灵活、精确控制，克服了传统方法依赖单一提示的局限。
摘要：
可控语音生成方法通常依赖单一或固定提示，阻碍了创造性与灵活性。这些限制使其难以满足特定场景下的用户需求，例如在保留所选说话人音色的同时调整风格，或选择风格并生成与角色视觉外观匹配的语音。为克服这些挑战，我们提出了FleSpeech，一种新颖的多阶段语音生成框架，通过整合各种形式的控制来实现更灵活的语音属性操纵。FleSpeech采用多模态提示编码器，将不同的文本、音频和视觉提示处理并统一为内聚表示。该方法增强了语音合成的适应性，支持对生成语音进行创造性且精确的控制。此外，我们开发了针对多模态数据集的数据收集流程，以促进该领域的进一步研究与应用。全面的主客观实验证明了FleSpeech的有效性。音频样本可在https://kkksuper.github.io/FleSpeech/获取。
英文摘要：
Controllable speech generation methods typically rely on single or fixed prompts, hindering creativity and flexibility. These limitations make it difficult to meet specific user needs in certain scenarios, such as adjusting the style while preserving a selected speaker's timbre, or choosing a style and generating a voice that matches a character's visual appearance. To overcome these challenges, we propose \textit{FleSpeech}, a novel multi-stage speech generation framework that allows for more flexible manipulation of speech attributes by integrating various forms of control. FleSpeech employs a multimodal prompt encoder that processes and unifies different text, audio, and visual prompts into a cohesive representation. This approach enhances the adaptability of speech synthesis and supports creative and precise control over the generated speech. Additionally, we develop a data collection pipeline for multimodal datasets to facilitate further research and applications in this field. Comprehensive subjective and objective experiments demonstrate the effectiveness of FleSpeech. Audio samples are available at https://kkksuper.github.io/FleSpeech/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
