Emotional voice conversion aims to convert the spectrum and prosody to change the emotional patterns of speech, while preserving the speaker identity and linguistic content. Many studies require parallel speech data between different emotional patterns, which is not practical in real life. Moreover, they often model the conversion of fundamental frequency (F0) with a simple linear transform. As F0 is a key aspect of intonation that is hierarchical in nature, we believe that it is more adequate to model F0 in different temporal scales by using wavelet transform. We propose a CycleGAN network to find an optimal pseudo pair from non-parallel training data by learning forward and inverse mappings simultaneously using adversarial and cycle-consistency losses. We also study the use of continuous wavelet transform (CWT) to decompose F0 into ten temporal scales, that describes speech prosody at different time resolution, for effective F0 conversion. Experimental results show that our proposed framework outperforms the baselines both in objective and subjective evaluations.
Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody
Graduate School of Artificial Intelligence（人工智能研究生院）；Dept. of Computer Science and Engineering（计算机科学与工程系）；Pohang University of Science and Technology (POSTECH)（釜山科学技术大学）
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Emotional voice conversion (EVC) aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity. In this paper, we study the disentanglement and recomposition of emotional elements in speech through variational autoencoding Wasserstein generative adversarial network (VAW-GAN). We propose a speaker-dependent EVC framework based on VAW-GAN, that includes two VAW-GAN pipelines, one for spectrum conversion, and another for prosody conversion. We train a spectral encoder that disentangles emotion and prosody (F0) information from spectra…
Northwestern Polytechnical University（西北工业大学）；Xi’an Jiaotong University（西安交通大学）；Mobvoi AI Lab（出门问问人工智能实验室）
Cross-speaker emotion transfer speech synthesis aims to synthesize emotional speech for a target speaker by transferring the emotion from reference speech recorded by another (source) speaker. In this task, extracting speaker-independent emotion embedding from reference speech plays an important role. However, the emotional information conveyed by such emotion embedding tends to be weakened in the process to squeeze out the source speaker's timbre information. In response to this problem, a prosody compensation module (PCM) is proposed in this paper to compensate for the emotional information …
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Emotional voice conversion aims to transform emotional prosody in speech while preserving the linguistic content and speaker identity. Prior studies show that it is possible to disentangle emotional prosody using an encoder-decoder network conditioned on discrete representation, such as one-hot emotion labels. Such networks learn to remember a fixed set of emotional styles. In this paper, we propose a novel framework based on variational auto-encoding Wasserstein generative adversarial network (VAW-GAN), which makes use of a pre-trained speech emotion recognition (SER) model to transfer emotio…
Machine-generated speech is characterized by its limited or unnatural emotional variation. Current text to speech systems generates speech with either a flat emotion, emotion selected from a predefined set, average variation learned from prosody sequences in training data or transferred from a source style. We propose a text to speech(TTS) system, where a user can choose the emotion of generated speech from a continuous and meaningful emotion space (Arousal-Valence space). The proposed TTS system can generate speech from the text in any speaker's style, with fine control of emotion. We show th…
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Emotional voice conversion aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity. The prior studies on emotional voice conversion are mostly carried out under the assumption that emotion is speaker-dependent. We consider that there is a common code between speakers for emotional expression in a spoken language, therefore, a speaker-independent mapping between emotional states is possible. In this paper, we propose a speaker-independent emotional voice conversion framework, that can convert anyone's emotion without the need …
We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embeddin…
PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts
Huawei Translation Service Center（华为翻译服务中心）；Key Laboratory of Child Development and Learning Science (Southeast University)（儿童发展与学习科学重点实验室（东南大学））；School of Biological Science and Medical Engineering（生物科学与医学工程学院）
Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion
Shanghai Jiao Tong University（上海交通大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National Institute of Informatics（信息学研究所）
LipSody: Lip-to-Speech Synthesis with Enhanced Prosody Consistency
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能跨学科项目，首尔国立大学）；Artificial Intelligence Institute, Seoul National University, Republic of Korea（人工智能研究所，首尔国立大学，韩国）
Emotional voice conversion models adapt the emotion in speech without changing the speaker identity or linguistic content. They are less data hungry than text-to-speech models and allow to generate large amounts of emotional data for downstream tasks. In this work we propose EmoCat, a language-agnostic emotional voice conversion model. It achieves high-quality emotion conversion in German with less than 45 minutes of German emotional recordings by exploiting large amounts of emotional data in US English. EmoCat is an encoder-decoder model based on CopyCat, a voice conversion system which trans…
School of Electrical Engineering and Informatics, Institut Teknologi Bandung（万隆理工学院电气工程与信息学院）
Adding an emotions using prosody manipulation method for Indonesian text to speech system. Text To Speech (TTS) is a system that can convert text in one language into speech, accordance with the reading of the text in the language used. The focus of this research is a natural sounding concept, the make "humanize" for the pronunciation of voice synthesis system Text To Speech. Humans have emotions / intonation that may affect the sound produced. The main requirement for the system used Text To Speech in this research is eSpeak, the database MBROLA using id1, Human Speech Corpus database from a …
Task-Vector Arithmetic for Emotional Expressivity Control in Language-Model-Based Text-to-Speech
Instituto de Biociências, Letras e Ciências Exatas Universidade Estadual Paulista "Júlio de Mesquita Filho" (UNESP)（生物科学、文学和精确科学学院 帕尔马斯州立大学 "Júlio de Mesquita Filho" (UNESP)）
HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech
South China University of Technology, China（华南理工大学，中国）；Huya Inc., China（欢丫公司，中国）；Tongyi Fun Team, Alibaba Group, China（通义 Fun 团队，阿里巴巴集团，中国）；Foshan University, China（佛山大学，中国）
Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling
Inner Mongolian University（内蒙古大学）；ByteDance（字节跳动）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National University of Singapore（新加坡国立大学）
Speech synthesis is used in a wide variety of industries. Nonetheless, it always sounds flat or robotic. The state of the art methods that allow for prosody control are very cumbersome to use and do not allow easy tuning. To tackle some of these drawbacks, in this work we target the implementation of a text-to-speech model where the inferred speech can be tuned with the desired emotions. To do so, we use Generative Adversarial Networks (GANs) together with a sequence-to-sequence model using an attention mechanism. We evaluate four different configurations considering different inputs and train…
Northwestern Polytechnical University（西北工业大学）；Tencent AI Lab（腾讯人工智能实验室）
Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specificall…
Centre for Speech Technology Research, University of Edinburgh（爱丁堡大学语音技术研究中心）
In English, prosody adds a broad range of information to segment sequences, from information structure (e.g. contrast) to stylistic variation (e.g. expression of emotion). However, when learning to control prosody in text-to-speech voices, it is not clear what exactly the control is modifying. Existing research on discrete representation learning for prosody has demonstrated high naturalness, but no analysis has been performed on what these representations capture, or if they can generate meaningfully-distinct variants of an utterance. We present a phrase-level variational autoencoder with a m…
Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models (i.e., training acoustic and vocoder models separately). However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as p…
ATRIE: Adaptive Tuning for Robust Inference and Emotion in Persona-Driven Speech Synthesis
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）；Shenzhen Polytechnic University（深圳职业技术大学）；University of Macau（澳门大学）；Huizhou University（惠州市大学）
CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
The University of Melbourne, Australia（墨尔本大学）；Wuhan University, China（武汉大学）；The Hong Kong University of Science（香港科学与技术大学）；The University of Auckland, New Zealand（奥克兰大学）；Monash University, Australia（莫纳什大学）
Imperial College London（伦敦帝国理工学院）；The University of Tokyo（东京大学）；University of Augsburg（奥格斯堡大学）
Emotional Voice Conversion (EVC) aims to convert the emotional style of a source speech signal to a target style while preserving its content and speaker identity information. Previous emotional conversion studies do not disentangle emotional information from emotion-independent information that should be preserved, thus transforming it all in a monolithic manner and generating audio of low quality, with linguistic distortions. To address this distortion problem, we propose a novel StarGAN framework along with a two-stage training process that separates emotional features from those independen…
NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
Department of Electrical and Computer Engineering, University of Texas at Dallas（德克萨斯大学达拉斯分校电气与计算机工程系）；Department of Electrical and Computer Engineering, Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系）；ARRAY Innovation（ARRAY创新）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
The cross-speaker emotion transfer task in text-to-speech (TTS) synthesis particularly aims to synthesize speech for a target speaker with the emotion transferred from reference speech recorded by another (source) speaker. During the emotion transfer process, the identity information of the source speaker could also affect the synthesized results, resulting in the issue of speaker leakage. This paper proposes a new method with the aim to synthesize controllable emotional expressive speech and meanwhile maintain the target speaker's identity in the cross-speaker emotion TTS task. The proposed m…
Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptation
School of Computer Science and Engineering, UNSW Sydney（新南威尔士大学悉尼分校计算机科学与工程学院）；School of Engineering, Institute of Science Tokyo（东京科学大学工程学院）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61部门）
Audiovisual speech synthesis is the problem of synthesizing a talking face while maximizing the coherency of the acoustic and visual speech. In this paper, we propose and compare two audiovisual speech synthesis systems for 3D face models. The first system is the AVTacotron2, which is an end-to-end text-to-audiovisual speech synthesizer based on the Tacotron2 architecture. AVTacotron2 converts a sequence of phonemes representing the sentence to synthesize into a sequence of acoustic features and the corresponding controllers of a face model. The output acoustic features are used to condition a…
Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing
Hangzhou Dianzi University（杭州电子科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Adelaide（阿德莱德大学）；Macquarie University（麦考瑞大学）
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
University of YYY（YYY大学）；Institute of WWW（WWW研究所）；The Hong Kong University of Science and Technology（香港科技大学）；Northwestern Polytechnical University（西北工业大学）；University of Science and Technology Beijing（北京科技大学）；Hong Kong Baptist University（香港浸会大学）；Chinese University of Hong Kong（香港中文大学）；Shanghai Mobvoi Information Technology Co., Ltd（上海小问信息技术有限公司）；University of Rochester（罗切斯特大学）；University of Surrey（萨里大学）
