We address the problem of human-in-the-loop control for generating prosody in the context of text-to-speech synthesis. Controlling prosody is challenging because existing generative models lack an efficient interface through which users can modify the output quickly and precisely. To solve this, we introduce a novel framework whereby the user provides partial inputs and the generative model generates the missing features. We propose a model that is specifically designed to encode partial prosodic features and output complete audio. We show empirically that our model displays two essential qualities of a human-in-the-loop control mechanism: efficiency and robustness. With even a very small number of input values (~4), our model enables users to improve the quality of the output significantly in terms of listener preference (4:1).
Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody
Graduate School of Artificial Intelligence(人工智能研究生院);Dept. of Computer Science and Engineering(计算机科学与工程系);Pohang University of Science and Technology (POSTECH)(釜山科学技术大学)
PoeticTTS -- Controllable Poetry Reading for Literary Studies
Institute for Natural Language Processing (IMS), University of Stuttgart(斯图加特大学自然语言处理研究所(IMS));Institute of Literary Studies (ILS), University of Stuttgart(斯图加特大学文学研究所(ILS));German Literature Archive (DLA)(德国文学档案馆(DLA))
DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Center for Language and Speech Processing, Johns Hopkins University, USA(语言与语音处理中心,约翰霍普金斯大学,美国);Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA(人类语言技术卓越中心(COE),约翰霍普金斯大学,美国)
Inner Mongolia University(内蒙古大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Bremen(不来梅大学)
PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts
Huawei Translation Service Center(华为翻译服务中心);Key Laboratory of Child Development and Learning Science (Southeast University)(儿童发展与学习科学重点实验室(东南大学));School of Biological Science and Medical Engineering(生物科学与医学工程学院)
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Macquarie University(麦觉里大学);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);University of Adelaide(阿德莱德大学)