arXiv 2310.060722023-10-09cs.SDeess.AS

JVNV：一个包含言语内容与非言语表达的日语情感语音语料库

JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions

Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；National Institute of Informatics（信息学研究所）

Detai Xin, Junfeng Jiang, Shinnosuke Takamichi, Yuki Saito, Akiko Aizawa, Hiroshi Saruwatari

DOI：: 10.1109/ACCESS.2024.3360885

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出首个用大语言模型自动生成脚本的日语情感语音语料库JVNV，包含言语内容与非言语发声，经筛选得到514个音素覆盖均衡的脚本，在音素覆盖度、情感可识别性上优于现有语料库，为情感语音合成带来新挑战与宝贵资源。

摘要：

我们提出了JVNV，这是一个包含言语内容与非言语发声的日语情感语音语料库，其脚本由大规模语言模型生成。现有情感语音语料库不仅缺乏合适的情感脚本，还缺少非言语发声（NVs）——这是口语中表达情感的关键表达方式。我们提出了一种自动脚本生成方法，通过向ChatGPT提供带有情感极性的种子词以及非言语发声短语，利用提示工程生成情感脚本。我们借助情感置信度得分和语言流畅度得分，从生成的候选脚本中选出514个音素覆盖均衡的脚本。我们验证了JVNV的有效性，结果表明JVNV比以往的日语情感语音语料库拥有更好的音素覆盖度和情感可识别性。随后我们使用离散编码表示NVs，在情感语音合成任务上对JVNV进行基准测试。我们发现，朗读语音合成与情感语音合成的性能之间仍存在差距，而在语音中加入NVs会让这项任务更具难度，这为该任务带来了新的挑战，也让JVNV成为未来相关工作的宝贵资源。据我们所知，JVNV是首个利用大规模语言模型自动生成脚本的语音语料库。

英文摘要：

We present the JVNV, a Japanese emotional speech corpus with verbal content and nonverbal vocalizations whose scripts are generated by a large-scale language model. Existing emotional speech corpora lack not only proper emotional scripts but also nonverbal vocalizations (NVs) that are essential expressions in spoken language to express emotions. We propose an automatic script generation method to produce emotional scripts by providing seed words with sentiment polarity and phrases of nonverbal vocalizations to ChatGPT using prompt engineering. We select 514 scripts with balanced phoneme coverage from the generated candidate scripts with the assistance of emotion confidence scores and language fluency scores. We demonstrate the effectiveness of JVNV by showing that JVNV has better phoneme coverage and emotion recognizability than previous Japanese emotional speech corpora. We then benchmark JVNV on emotional text-to-speech synthesis using discrete codes to represent NVs. We show that there still exists a gap between the performance of synthesizing read-aloud speech and emotional speech, and adding NVs in the speech makes the task even harder, which brings new challenges for this task and makes JVNV a valuable resource for relevant works in the future. To our best knowledge, JVNV is the first speech corpus that generates scripts automatically using large language models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。