Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models (i.e., training acoustic and vocoder models separately). However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as pitch and voicing flags, from the input text. From these features, the proposed periodicity generator produces a sample-level sinusoidal source that enables the waveform decoder to accurately reproduce the pitch. Finally, the entire model is jointly optimized in an end-to-end manner with variational inference and adversarial objectives. As a result, the decoder becomes capable of generating more stable, expressive, and natural output waveforms. The experimental results showed that the proposed model significantly outperforms baseline models in terms of naturalness, with improved pitch stability in the generated samples.
Data augmentation via voice conversion (VC) has been successfully applied to low-resource expressive text-to-speech (TTS) when only neutral data for the target speaker are available. Although the quality of VC is crucial for this approach, it is challenging to learn a stable VC model because the amount of data is limited in low-resource scenarios, and highly expressive speech has large acoustic variety. To address this issue, we propose a novel data augmentation method that combines pitch-shifting and VC techniques. Because pitch-shift data augmentation enables the coverage of a variety of pit…
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Expressive voice conversion performs identity conversion for emotional speakers by jointly converting speaker identity and emotional style. Due to the hierarchical structure of speech emotion, it is challenging to disentangle the emotional style for different speakers. Inspired by the recent success of speaker disentanglement with variational autoencoder (VAE), we propose an any-to-any expressive voice conversion framework, that is called StyleVC. StyleVC is designed to disentangle linguistic content, speaker identity, pitch, and emotional style information. We study the use of style encoder t…
Osaka University（大阪大学）；Curtin University（科廷大学）；National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；CyberAgent, Inc.（赛博代理公司）
Emotional voice conversion (VC) aims to convert a neutral voice to an emotional (e.g. happy) one while retaining the linguistic information and speaker identity. We note that the decoupling of emotional features from other speech information (such as speaker, content, etc.) is the key to achieving remarkable performance. Some recent attempts about speech representation decoupling on the neutral speech can not work well on the emotional speech, due to the more complex acoustic properties involved in the latter. To address this problem, here we propose a novel Source-Filter-based Emotional VC mo…
We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embeddin…
The way that humans encode their emotion into speech signals is complex. For instance, an angry man may increase his pitch and speaking rate, and use impolite words. In this paper, we present a preliminary study on various emotional factors and investigate how each of them impacts modern emotion recognition systems. The key tool of our study is the SpeechFlow model presented recently, by which we are able to decompose speech signals into separate information factors (content, pitch, rhythm). Based on this decomposition, we carefully studied the performance of each information component and the…
School of Electrical Engineering and Informatics, Institut Teknologi Bandung（万隆理工学院电气工程与信息学院）
Adding an emotions using prosody manipulation method for Indonesian text to speech system. Text To Speech (TTS) is a system that can convert text in one language into speech, accordance with the reading of the text in the language used. The focus of this research is a natural sounding concept, the make "humanize" for the pronunciation of voice synthesis system Text To Speech. Humans have emotions / intonation that may affect the sound produced. The main requirement for the system used Text To Speech in this research is eSpeak, the database MBROLA using id1, Human Speech Corpus database from a …
Due to improvements in artificial intelligence, speaker identification (SI) technologies have brought a great direction and are now widely used in a variety of sectors. One of the most important components of SI is feature extraction, which has a substantial impact on the SI process and performance. As a result, numerous feature extraction strategies are thoroughly investigated, contrasted, and analyzed. This article exploits five distinct feature extraction methods for speaker identification in disguised voices under emotional environments. To evaluate this work significantly, three effects a…
Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTS
Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部认知计算与应用天津市重点实验室）；Kuaishou Technology Co., Ltd（快手科技有限公司）；Institute of Artificial Intelligence, China Telecom（中国电信人工智能研究院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Huiyan Technology (Tianjin) Co., Ltd（慧眼科技（天津）有限公司）
The cross-speaker emotion transfer task in text-to-speech (TTS) synthesis particularly aims to synthesize speech for a target speaker with the emotion transferred from reference speech recorded by another (source) speaker. During the emotion transfer process, the identity information of the source speaker could also affect the synthesized results, resulting in the issue of speaker leakage. This paper proposes a new method with the aim to synthesize controllable emotional expressive speech and meanwhile maintain the target speaker's identity in the cross-speaker emotion TTS task. The proposed m…
LipSody: Lip-to-Speech Synthesis with Enhanced Prosody Consistency
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能跨学科项目，首尔国立大学）；Artificial Intelligence Institute, Seoul National University, Republic of Korea（人工智能研究所，首尔国立大学，韩国）
SponTTS: modeling and transferring spontaneous style for TTS
Northwestern Polytechnical University（西北工业大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Mobvoi Information Technology Co., Ltd（上海出门问问信息技术有限公司）
Pitch or fundamental frequency (f0) extraction is a fundamental problem studied extensively for its potential applications in speech and clinical applications. In literature, explicit mode specific (modal speech or singing voice or emotional/ expressive speech or noisy speech) signal processing and deep learning f0 extraction methods that exploit the quasi periodic nature of the signal in time, harmonic property in spectral or combined form to extract the pitch is developed. Hence, there is no single unified method which can reliably extract the pitch from various modes of the acoustic signal.…
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation（日本电信电话株式会社 NTT 通信科学研究所）
This paper proposes a non-autoregressive extension of our previously proposed sequence-to-sequence (S2S) model-based voice conversion (VC) methods. S2S model-based VC methods have attracted particular attention in recent years for their flexibility in converting not only the voice identity but also the pitch contour and local duration of input speech, thanks to the ability of the encoder-decoder architecture with the attention mechanism. However, one of the obstacles to making these methods work in real-time is the autoregressive (AR) structure. To overcome this obstacle, we develop a method t…
Comparative Analysis of Mel-Frequency Cepstral Coefficients and Wavelet Based Audio Signal Processing for Emotion Detection and Mental Health Assessment in Spoken Speech
Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms
University of Information Technology & Sciences（信息技术与科学大学）；Southeast University（东南大学）；Endicott College, Woosong University（又松大学恩迪科特学院）；Daffodil International University（水仙国际大学）；Jahangirnagar University（贾汉吉尔纳加尔大学）
School of Electronics Engineering, Vellore Institute of Technology（韦洛尔理工大学电子工程学院）
In this project, we aim to classify the speech taken as one of the four emotions namely, sadness, anger, fear and happiness. The samples that have been taken to complete this project are taken from Linguistic Data Consortium (LDC) and UGA database. The important characteristics determined from the samples are energy, pitch, MFCC coefficients, LPCC coefficients and speaker rate. The classifier used to classify these emotional states is Support Vector Machine (SVM) and this is done using two classification strategies: One against All (OAA) and Gender Dependent Classification. Furthermore, a comp…
In this work, we conduct an extensive comparison of various approaches to speech based emotion recognition systems. The analyses were carried out on audio recordings from Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). After pre-processing the raw audio files, features such as Log-Mel Spectrogram, Mel-Frequency Cepstral Coefficients (MFCCs), pitch and energy were considered. The significance of these features for emotion classification was compared by applying methods such as Long Short Term Memory (LSTM), Convolutional Neural Networks (CNNs), Hidden Markov Models (HMMs) …
National Institute of Technology Patna（巴特那国家技术学院）；Central University of South Bihar（南比哈尔中央大学）
Speech is produced when time varying vocal tract system is excited with time varying excitation source. Therefore, the information present in a speech such as message, emotion, language, speaker is due to the combined effect of both excitation source and vocal tract system. However, there is very less utilization of excitation source features to recognize emotion. In our earlier work, we have proposed a novel method to extract glottal closure instants (GCIs) known as epochs. In this paper, we have explored epoch features namely instantaneous pitch, phase and strength of epochs for discriminati…
SuZhou Automotive Research Institute of Tsinghua University（清华大学苏州汽车研究院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Hyundai Motor Advanced Tech. R&D Center；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）
Speech Emotion Recognition Using MFCC Features and LSTM-Based Deep Learning Model
Department of Electrical Electronics and Telecommunications Engineering, Bells University of Technology（贝尔大学技术学院电子与电信工程系）；New Horizons Bells University of Technology（新希望贝尔大学技术学院）；Department of Computer Engineering, Bells University of Technology（贝尔大学技术学院计算机工程系）；Department of Computer and Information Sciences, Towson University（托逊大学计算机与信息科学系）
