This paper describes progress towards making a Neural Text-to-Speech (TTS) Frontend that works for many languages and can be easily extended to new languages. We take a Machine Translation (MT) inspired approach to constructing the frontend, and model both text normalization and pronunciation on a sentence level by building and using sequence-to-sequence (S2S) models. We experimented with training normalization and pronunciation as separate S2S models and with training a single S2S model combining both functions.
For our language-independent approach to pronunciation we do not use a lexicon. Instead all pronunciations, including context-based pronunciations, are captured in the S2S model. We also present a language-independent chunking and splicing technique that allows us to process arbitrary-length sentences. Models for 18 languages were trained and evaluated. Many of the accuracy measurements are above 99%. We also evaluated the models in the context of end-to-end synthesis against our current production system.
Chinese dialects are different variations of Chinese and can be considered as different languages in the same language family with Mandarin. Though they all use Chinese characters, the pronunciations, grammar and idioms can vary significantly, and even local speakers may find it hard to input correct written forms of dialect. Besides, using Mandarin text as text-to-speech inputs would generate speech with poor naturalness. In this paper, we propose a novel Chinese dialect TTS frontend with a translation module, which converts Mandarin text into dialectic expressions to improve the intelligibil…
An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves be…
Amazon Research（亚马逊研究院）；Warsaw University of Technology（华沙理工大学）
This work focuses on modelling a speaker's accent that does not have a dedicated text-to-speech (TTS) frontend, including a grapheme-to-phoneme (G2P) module. Prior work on modelling accents assumes a phonetic transcription is available for the target accent, which might not be the case for low-resource, regional accents. In our work, we propose an approach whereby we first augment the target accent data to sound like the donor voice via voice conversion, then train a multi-speaker multi-accent TTS model on the combination of recordings and synthetic data, to generate the donor's voice speaking…
Bolbosh: Script-Aware Flow Matching for Kashmiri Text-to-Speech
King Abdullah University of Science and Technology (KAUST)（卡塔尔国王 Abdullah 科学与技术大学）；Department of Linguistics, University of Kashmir（克什米尔大学语言学系）；National Institute of Technology Srinagar（锡里纳加尔国家理工学院）
Microsoft Research India（微软印度研究院）；Microsoft Corporation（微软公司）
Spoken Term Detection (STD) is the task of searching for words or phrases within audio, given either text or spoken input as a query. In this work, we use state-of-the-art Hindi, Tamil and Telugu ASR systems cross-lingually for lexical Spoken Term Detection in ten low-resource Indian languages. Since no publicly available dataset exists for Spoken Term Detection in these languages, we create a new dataset using a publicly available TTS dataset. We report a standard metric for STD, Mean Term Weighted Value (MTWV) and show that ASR systems built in languages that are phonetically similar to the …
Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data
Signal Analysis and Interpretation Lab, University of Southern California, USA（信号分析与解释实验室，南加州大学）；Thomas Lord Department of Computer Science, University of Southern California, USA（托马斯·劳德计算机科学系，南加州大学）；Department of Linguistics, University of Southern California（语言学系，南加州大学）
This paper proposes Virtuoso, a massively multilingual speech-text joint semi-supervised learning framework for text-to-speech synthesis (TTS) models. Existing multilingual TTS typically supports tens of languages, which are a small fraction of the thousands of languages in the world. One difficulty to scale multilingual TTS to hundreds of languages is collecting high-quality speech-text paired data in low-resource languages. This study extends Maestro, a speech-text joint pretraining framework for automatic speech recognition (ASR), to speech generation tasks. To train a TTS model from variou…
While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the…
We present a model of text analysis for text-to-speech (TTS) synthesis based on (weighted) finite-state transducers, which serves as the text-analysis module of the multilingual Bell Labs TTS system. The transducers are constructed using a lexical toolkit that allows declarative descriptions of lexicons, morphological rules, numeral-expansion rules, and phonological rules, inter alia. To date, the model has been applied to eight languages: Spanish, Italian, Romanian, French, German, Russian, Mandarin and Japanese.
Towards a Phonology-Informed Evaluation of Multilingual TTS
Centre for Linguistic Science & Technology, IIT Guwahati（印度理工学院古瓦哈提分校语言科学与技术中心）；Mehta Family School for Data Science & Artificial Intelligence, IIT Guwahati（印度理工学院古瓦哈提分校梅塔家族数据科学与人工智能学院）；Department of Humanities & Social Sciences, IIT Guwahati（印度理工学院古瓦哈提分校人文与社会科学系）
The idea of using phonological features instead of phonemes as input to sequence-to-sequence TTS has been recently proposed for zero-shot multilingual speech synthesis. This approach is useful for code-switching, as it facilitates the seamless uttering of foreign text embedded in a stream of native text. In our work, we train a language-agnostic multispeaker model conditioned on a set of phonologically derived features common across different languages, with the goal of achieving cross-lingual speaker adaptation. We first experiment with the effect of language phonological similarity on cross-…
Generalized Multilingual Text-to-Speech Generation with Language-Aware Style Adaptation
School of Computer Science and Engineering, UNSW Sydney（新南威尔士大学悉尼分校计算机科学与工程学院）；School of Engineering, Institute of Science Tokyo（东京科学大学工程学院）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61部门）
A multilingual training strategy for low resource Text to Speech
ENSIAS, Mohammed V University in Rabat（拉巴特穆罕默德五世大学ENSIAS学院）；International University of Rabat（拉巴特国际大学）；University of Leeds（利兹大学）；Sorbonne Université（索邦大学）
In this paper, we propose GlowVC: a multilingual multi-speaker flow-based model for language-independent text-free voice conversion. We build on Glow-TTS, which provides an architecture that enables use of linguistic features during training without the necessity of using them for VC inference. We consider two versions of our model: GlowVC-conditional and GlowVC-explicit. GlowVC-conditional models the distribution of mel-spectrograms with speaker-conditioned flow and disentangles the mel-spectrogram space into content- and pitch-relevant dimensions, while GlowVC-explicit models the explicit di…
OpenBibleTTS: Large-Scale Speech Resources and TTS Models for Low-Resource Languages
McGill University（麦吉尔大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；AIMS Research and Innovation Centre（AIMS研究与创新中心）；NM-AIST；Saarland University（萨尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能教席）
An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；National Institute of Informatics（日本信息学研究所）；National Research Council Canada（加拿大国家研究委员会）；The Centre for Speech Technology Research, University of Edinburgh（爱丁堡大学言语技术研究中心）；National Institute of Information and Communications Technology（日本信息通信研究机构）
The CMU Wilderness Multilingual Speech Dataset (Black, 2019) is a newly published multilingual speech dataset based on recorded readings of the New Testament. It provides data to build Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models for potentially 700 languages. However, the fact that the source content (the Bible) is the same for all the languages is not exploited to date.Therefore, this article proposes to add multilingual links between speech segments in different languages, and shares a large and clean dataset of 8,130 parallel spoken utterances across 8 languages (56 l…
University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Science and Technology of China（中国科学技术大学）
This work presents a lifelong learning approach to train a multilingual Text-To-Speech (TTS) system, where each language was seen as an individual task and was learned sequentially and continually. It does not require pooled data from all languages altogether, and thus alleviates the storage and computation burden. One of the challenges of lifelong learning methods is "catastrophic forgetting": in TTS scenario it means that model performance quickly degrades on previous languages when adapted to a new language. We approach this problem via a data-replay-based lifelong learning method. We formu…
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
College of Computing and Data Science（计算与数据科学学院）；Nanyang Technological University（南洋理工大学）；AI Singapore（AI新加坡）；National University of Singapore（新加坡国立大学）；Institute for Infocomm Research (I$^2$R)（信息通信研究所）；A$^{*}$STAR（A*星）
Training multilingual Neural Text-To-Speech (NTTS) models using only monolingual corpora has emerged as a popular way for building voice cloning based Polyglot NTTS systems. In order to train these models, it is essential to understand how the composition of the training corpora affects the quality of multilingual speech synthesis. In this context, it is common to hear questions such as "Would including more Spanish data help my Italian synthesis, given the closeness of both languages?". Unfortunately, we found existing literature on the topic lacking in completeness in this regard. In the pre…
This paper introduces Multilingual LibriSpeech (MLS) dataset, a large multilingual corpus suitable for speech research. The dataset is derived from read audiobooks from LibriVox and consists of 8 languages, including about 44.5K hours of English and a total of about 6K hours for other languages. Additionally, we provide Language Models (LM) and baseline Automatic Speech Recognition (ASR) models and for all the languages in our dataset. We believe such a large transcribed dataset will open new avenues in ASR and Text-To-Speech (TTS) research. The dataset will be made freely available for anyone…
