Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.
Generally speaking, the main objective when training a neural speech synthesis system is to synthesize natural and expressive speech from the output layer of the neural network without much attention given to the hidden layers. However, by learning useful latent representation, the system can be used for many more practical scenarios. In this paper, we investigate the use of quantized vectors to model the latent linguistic embedding and compare it with the continuous counterpart. By enforcing different policies over the latent spaces in the training, we are able to obtain a latent linguistic e…
Chinese Academy of Science（中国科学院）；Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）
Text-to-speech (TTS) and voice conversion (VC) are two different tasks both aiming at generating high quality speaking voice according to different input modality. Due to their similarity, this paper proposes UnifySpeech, which brings TTS and VC into a unified framework for the first time. The model is based on the assumption that speech can be decoupled into three independent components: content information, speaker information, prosody information. Both TTS and VC can be regarded as mining these three parts of information from the input and completing the reconstruction of speech. For TTS, t…
University of Manchester（曼彻斯特大学）；University of Cambridge（剑桥大学）
Text-to-speech (TTS) systems offer the opportunity to compensate for a hearing loss at the source rather than correcting for it at the receiving end. This removes limitations such as time constraints for algorithms that amplify a sound in a hearing aid and can lead to higher speech quality. We propose an algorithm that restores loudness to normal perception at a high resolution in time, frequency and level, and embed it in a TTS system that uses Tacotron2 and WaveGlow to produce individually amplified speech. Subjective evaluations of speech quality showed that the proposed algorithm led to hi…
UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information
International Digital Economy Academy（国际数字经济学院）；IDEA&Emdoor Collaborative Laboratory（IDEA与Emdoor协同实验室）；Emdoor Information Co., Ltd（Emdoor信息有限公司）；Shenzhen Yijiayiban Information Technology Co., Ltd（深圳艺佳iban信息技术有限公司）
Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation
Graduate School of Artificial Intelligence, POSTECH（浦项工科大学人工智能研究生院）；Department of Computer Science and Engineering, POSTECH（浦项工科大学计算机科学与工程系）；aiXplain Inc.（aiXplain公司）
Numediart Institute University of Mons（蒙斯大学Numediart研究所）
In this project, we aim to build a Text-to-Speech system able to produce speech with a controllable emotional expressiveness. We propose a methodology for solving this problem in three main steps. The first is the collection of emotional speech data. We discuss the various formats of existing datasets and their usability in speech generation. The second step is the development of a system to automatically annotate data with emotion/expressiveness features. We compare several techniques using transfer learning to extract such a representation through other tasks and propose a method to visualiz…
Recently, sequence-to-sequence models with attention have been successfully applied in Text-to-speech (TTS). These models can generate near-human speech with a large accurately-transcribed speech corpus. However, preparing such a large data-set is both expensive and laborious. To alleviate the problem of heavy data demand, we propose a novel unsupervised pre-training mechanism in this paper. Specifically, we first use Vector-quantization Variational-Autoencoder (VQ-VAE) to ex-tract the unsupervised linguistic units from large-scale, publicly found, and untranscribed speech. We then pre-train t…
Faculty of Applied Sciences（应用科学学院）；School of Computing Science（计算科学学院）；FEMTO-ST Institute Centre national de la recherche scientifique (CNRS)（FEMTO-ST研究所，国家科学研究中心）；Centre national de la recherche scientifique (CNRS)（国家科学研究中心）
Generative Semantic Communication for Text-to-Speech Synthesis
Shenzhen Future Network of Intelligence Institute（深圳未来网络智能研究院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen)（广东省未来网络智能重点实验室，香港中文大学（深圳））；School of Automation, Central South University（中南大学自动化学院）
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Microsoft Research（微软研究院）；Microsoft Azure（微软Azure）；University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；The University of Tokyo（东京大学）；Peking University（北京大学）
Zapr Media Labs (Red Brick Lane Marketing Solutions Pvt. Ltd.)（Zapr媒体实验室（红砖巷营销解决方案私人有限公司））
The perceptual quality of neural text-to-speech (TTS) is highly dependent on the choice of the model during training. Selecting the model using a training-objective metric such as the least mean squared error does not always correlate with human perception. In this paper, we propose an objective metric based on the phone error rate (PER) to select the TTS model with the best speech intelligibility. The PER is computed between the input text to the TTS model, and the text decoded from the synthesized speech using an automatic speech recognition (ASR) model, which is trained on the same data as …
School of Electrical Engineering and Informatics, Institut Teknologi Bandung（万隆理工学院电气工程与信息学院）
Adding an emotions using prosody manipulation method for Indonesian text to speech system. Text To Speech (TTS) is a system that can convert text in one language into speech, accordance with the reading of the text in the language used. The focus of this research is a natural sounding concept, the make "humanize" for the pronunciation of voice synthesis system Text To Speech. Humans have emotions / intonation that may affect the sound produced. The main requirement for the system used Text To Speech in this research is eSpeak, the database MBROLA using id1, Human Speech Corpus database from a …
Institute for Natural Language Processing (IMS), University of Stuttgart（斯图加特大学自然语言处理研究所（IMS））；Institute of Literary Studies (ILS), University of Stuttgart（斯图加特大学文学研究所（ILS））；German Literature Archive (DLA)（德国文学档案馆（DLA））
Speech synthesis for poetry is challenging due to specific intonation patterns inherent to poetic speech. In this work, we propose an approach to synthesise poems with almost human like naturalness in order to enable literary scholars to systematically examine hypotheses on the interplay between text, spoken realisation, and the listener's perception of poems. To meet these special requirements for literary studies, we resynthesise poems by cloning prosodic values from a human reference recitation, and afterwards make use of fine-grained prosody control to manipulate the synthetic speech in a …
Recent neural text-to-speech (TTS) models with fine-grained latent features enable precise control of the prosody of synthesized speech. Such models typically incorporate a fine-grained variational autoencoder (VAE) structure, extracting latent features at each input token (e.g., phonemes). However, generating samples with the standard VAE prior often results in unnatural and discontinuous speech, with dramatic prosodic variation between tokens. This paper proposes a sequential prior in a discrete latent space which can generate more naturally sounding samples. This is accomplished by discreti…
QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan（计算机科学与信息工程系，台湾国立台湾师范大学）；Institute of Computer Science, Academia Sinica, Taiwan（学术院计算机科学研究所，台湾）；United Link Co., Ltd., Taiwan（联合链接公司，台湾）
This paper describes the Microsoft end-to-end neural text to speech (TTS) system: DelightfulTTS for Blizzard Challenge 2021. The goal of this challenge is to synthesize natural and high-quality speech from text, and we approach this goal in two perspectives: The first is to directly model and generate waveform in 48 kHz sampling rate, which brings higher perception quality than previous systems with 16 kHz or 24 kHz sampling rate; The second is to model the variation information in speech through a systematic design, which improves the prosody and naturalness. Specifically, for 48 kHz modeling…
