Emotional voice conversion (EVC) involves modifying various acoustic characteristics, such as pitch and spectral envelope, to match a desired emotional state while preserving the speaker's identity. Existing EVC methods often rely on text transcriptions or time-alignment information and struggle to handle varying speech durations effectively. In this paper, we propose DurFlex-EVC, a duration-flexible EVC framework that operates without the need for text or alignment information. We introduce a unit aligner that models contextual information by aligning speech with discrete units representing content, eliminating the need for text or speech-text alignment. Additionally, we design a style autoencoder that effectively disentangles content and emotional style, allowing precise manipulation of the emotional characteristics of the speech. We further enhance emotional expressiveness through a hierarchical stylize encoder that applies the target emotional style at multiple hierarchical levels, refining the stylization process to improve the naturalness and expressiveness of the converted speech. Experimental results from subjective and objective evaluations demonstrate that our approach outperforms baseline models, effectively handling duration variability and enhancing emotional expressiveness in the converted speech.
Maestro-EVC: Controllable Emotional Voice Conversion Guided by References and Explicit Prosody
Graduate School of Artificial Intelligence（人工智能研究生院）；Dept. of Computer Science and Engineering（计算机科学与工程系）；Pohang University of Science and Technology (POSTECH)（釜山科学技术大学）
This paper shows that StarGAN-VC, a spectral envelope transformation method for non-parallel many-to-many voice conversion (VC), is capable of emotional VC (EVC). Although StarGAN-VC has been shown to enable speaker identity conversion, its capability for EVC for Japanese phrases has not been clarified. In this paper, we describe the direct application of StarGAN-VC to an EVC task with minimal fundamental frequency and aperiodicity processing. Through subjective evaluation experiments, we evaluated the performance of our StarGAN-EVC system in terms of its ability to achieve EVC for Japanese ph…
Mixed-EVC: Mixed Emotion Synthesis and Control in Voice Conversion
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；National University of Singapore（新加坡国立大学）
Graduate School of Engineering Science, Osaka University（大阪大学工学研究科）；Advanced Telecommunications Research Institute International（国际高级电信研究所）；RIKEN（理化学研究所）
In this study, we explore the transformer's ability to capture intra-relations among frames by augmenting the receptive field of models. Concretely, we propose a CycleGAN-based model with the transformer and investigate its ability in the emotional voice conversion task. In the training procedure, we adopt curriculum learning to gradually increase the frame length so that the model can see from the short segment till the entire speech. The proposed method was evaluated on the Japanese emotional speech dataset and compared to several baselines (ACVAE, CycleGAN) with objective and subjective eva…
University of Augsburg（奥格斯堡大学）；Huawei Technologies（华为技术有限公司）；Imperial College London（伦敦帝国理工学院）
Emotional voice conversion (EVC) focuses on converting a speech utterance from a source to a target emotion; it can thus be a key enabling technology for human-computer interaction applications and beyond. However, EVC remains an unsolved research problem with several challenges. In particular, as speech rate and rhythm are two key factors of emotional conversion, models have to generate output sequences of differing length. Sequence-to-sequence modelling is recently emerging as a competitive paradigm for models that can overcome those challenges. In an attempt to stimulate further research in…
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Southern Queensland（南昆士兰大学）；Imperial College London（帝国理工学院）；University of Augsburg（奥格斯堡大学）；University of Bremen（不来梅大学）
Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In EVC, emotions are usually treated as discrete categories overlooking the fact that speech also conveys emotions with various intensity levels that the listener can perceive. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.…
Towards Realistic Emotional Voice Conversion using Controllable Emotional Intensity
Key Laboratory of Child Development and Learning Science (Southeast University)（儿童发展与学习科学重点实验室（东南大学））；School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Emotional voice conversion (EVC) aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity. In this paper, we study the disentanglement and recomposition of emotional elements in speech through variational autoencoding Wasserstein generative adversarial network (VAW-GAN). We propose a speaker-dependent EVC framework based on VAW-GAN, that includes two VAW-GAN pipelines, one for spectrum conversion, and another for prosody conversion. We train a spectral encoder that disentangles emotion and prosody (F0) information from spectra…
PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts
Huawei Translation Service Center（华为翻译服务中心）；Key Laboratory of Child Development and Learning Science (Southeast University)（儿童发展与学习科学重点实验室（东南大学））；School of Biological Science and Medical Engineering（生物科学与医学工程学院）
Imperial College London（伦敦帝国理工学院）；The University of Tokyo（东京大学）；University of Augsburg（奥格斯堡大学）
Emotional Voice Conversion (EVC) aims to convert the emotional style of a source speech signal to a target style while preserving its content and speaker identity information. Previous emotional conversion studies do not disentangle emotional information from emotion-independent information that should be preserved, thus transforming it all in a monolithic manner and generating audio of low quality, with linguistic distortions. To address this distortion problem, we propose a novel StarGAN framework along with a two-stage training process that separates emotional features from those independen…
Sony Research India（索尼研究印度）；Sony Group Corporation, Japan（索尼集团公司）
Primary goal of an emotional voice conversion (EVC) system is to convert the emotion of a given speech signal from one style to another style without modifying the linguistic content of the signal. Most of the state-of-the-art approaches convert emotions for seen speaker-emotion combinations only. In this paper, we tackle the problem of converting the emotion of speakers whose only neutral data are present during the time of training and testing (i.e., unseen speaker-emotion combinations). To this end, we extend a recently proposed StartGANv2-VC architecture by utilizing dual encoders for lear…
Due to improvements in artificial intelligence, speaker identification (SI) technologies have brought a great direction and are now widely used in a variety of sectors. One of the most important components of SI is feature extraction, which has a substantial impact on the SI process and performance. As a result, numerous feature extraction strategies are thoroughly investigated, contrasted, and analyzed. This article exploits five distinct feature extraction methods for speaker identification in disguised voices under emotional environments. To evaluate this work significantly, three effects a…
