SpeechCraft:一个带有自然语言描述的细粒度表现性语音数据集
SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description
AI总结:
针对语音风格细微信息导致的多模态学习挑战,提出自动语音标注系统并构建了包含约2000小时音频的双语数据集SpeechCraft,显著提升了语音合成与风格理解的性能。
摘要:
语音风格中固有的细微信息使得语音-语言多模态学习面临重大挑战。因此,迫切需要一个能对语音风格进行详尽理解的大规模数据集,以促进语音音频与自然语言之间的深度交互。然而,构建此类数据集在大规模数据收集与高质量标注之间存在重大权衡。为应对这一挑战,我们提出了一种用于表现力解读的自动语音标注系统,该系统使用富有表现力且生动的人类语言描述对自然场景下的语音片段进行标注。最初,语音音频由一系列专家分类器和字幕模型处理以捕捉多样的语音特征,随后通过微调的LLaMA生成定制化标注。与以往基于标签/模板且信息和多样性有限的标注框架不同,我们的系统通过量身定制的自然语言描述提供对语音风格的深入理解,从而能够为大模型训练生成准确且海量的数据。借助该系统,我们创建了SpeechCraft,一个细粒度的双语表现性语音数据集。其显著特点是包含高度描述性的自然语言风格提示,包含约2,000小时的音频数据,涵盖超过两百万个语音片段。大量实验表明,所提出的数据集显著提升了语音-语言任务在风格化语音合成和语音风格理解方面的性能。
英文摘要:
Speech-language multi-modal learning presents a significant challenge due to the fine nuanced information inherent in speech styles. Therefore, a large-scale dataset providing elaborate comprehension of speech style is urgently needed to facilitate insightful interplay between speech audio and natural language. However, constructing such datasets presents a major trade-off between large-scale data collection and high-quality annotation. To tackle this challenge, we propose an automatic speech annotation system for expressiveness interpretation that annotates in-the-wild speech clips with expressive and vivid human language descriptions. Initially, speech audios are processed by a series of expert classifiers and captioning models to capture diverse speech characteristics, followed by a fine-tuned LLaMA for customized annotation generation. Unlike previous tag/templet-based annotation frameworks with limited information and diversity, our system provides in-depth understandings of speech style through tailored natural language descriptions, thereby enabling accurate and voluminous data generation for large model training. With this system, we create SpeechCraft, a fine-grained bilingual expressive speech dataset. It is distinguished by highly descriptive natural language style prompts, containing approximately 2,000 hours of audio data and encompassing over two million speech clips. Extensive experiments demonstrate that the proposed dataset significantly boosts speech-language task performance in stylist speech synthesis and speech style understanding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。