In accented voice conversion or accent conversion, we seek to convert the accent in speech from one another while preserving speaker identity and semantic content. In this study, we formulate a novel method for creating multi-accented speech samples, thus pairs of accented speech samples by the same speaker, through text transliteration for training accent conversion systems. We begin by generating transliterated text with Large Language Models (LLMs), which is then fed into multilingual TTS models to synthesize accented English speech. As a reference system, we built a sequence-to-sequence model on the synthetic parallel corpus for accent conversion. We validated the proposed method for both native and non-native English speakers. Subjective and objective evaluations further validate our dataset's effectiveness in accent conversion studies.
This paper proposes an interesting voice and accent joint conversion approach, which can convert an arbitrary source speaker's voice to a target speaker with non-native accent. This problem is challenging as each target speaker only has training data in native accent and we need to disentangle accent and speaker information in the conversion model training and re-combine them in the conversion stage. In our recognition-synthesis conversion framework, we manage to solve this problem by two proposed tricks. First, we use accent-dependent speech recognizers to obtain bottleneck features for diffe…
Xinjiang University（新疆大学）；Bytedance AI LAB（字节跳动人工智能实验室）
Accent conversion (AC) transforms a non-native speaker's accent into a native accent while maintaining the speaker's voice timbre. In this paper, we propose approaches to improving accent conversion applicability, as well as quality. First of all, we assume no reference speech is available at the conversion stage, and hence we employ an end-to-end text-to-speech system that is trained on native speech to generate native reference speech. To improve the quality and accent of the converted speech, we introduce reference encoders which make us capable of utilizing multi-source information. This i…
AccentBox: Towards High-Fidelity Zero-Shot Accent Generation
Centre for Speech Technology Research, University of Edinburgh, UK（语音技术研究中心，爱丁堡大学，英国）；Department of AI Technology, Transsion, China（人工智能技术系，Transsion，中国）
Amazon Research（亚马逊研究院）；Warsaw University of Technology（华沙理工大学）
This work focuses on modelling a speaker's accent that does not have a dedicated text-to-speech (TTS) frontend, including a grapheme-to-phoneme (G2P) module. Prior work on modelling accents assumes a phonetic transcription is available for the target accent, which might not be the case for low-resource, regional accents. In our work, we propose an approach whereby we first augment the target accent data to sound like the donor voice via voice conversion, then train a multi-speaker multi-accent TTS model on the combination of recordings and synthetic data, to generate the donor's voice speaking…
Voice conversion (VC) is a task that transforms the source speaker's timbre, accent, and tones in audio into another one's while preserving the linguistic content. It is still a challenging work, especially in a one-shot setting. Auto-encoder-based VC methods disentangle the speaker and the content in input speech without given the speaker's identity, so these methods can further generalize to unseen speakers. The disentangle capability is achieved by vector quantization (VQ), adversarial training, or instance normalization (IN). However, the imperfect disentanglement may harm the quality of o…
Accent Conversion: A Problem-Driven Survey of Sociolinguistic and Technical Constraints
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学与数据科学学院）；National Center for Supercomputing Applications, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校国家超级计算应用中心）；Department of East Asian Languages and Cultures, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校东亚语言与文化系）
Columbia University（哥伦比亚大学）；Recognition Technologies, Inc.（识别技术公司）
Two new approaches to accent classification and conversion are presented and explored, respectively. The first topic is Chinese accent classification/recognition. The second topic is the use of encoder-decoder models for end-to-end Chinese accent conversion, where the classifier in the first topic is used for the training of the accent converter encoder-decoder model. Experiments using different features and model are performed for accent recognition. These features include MFCCs and spectrograms. The classifier models were TDNN and 1D-CNN. On the MAGICDATA dataset with 5 classes of accents, t…
NTT Communication Science Laboratories（NTT通信科学实验室）；Nippon Telegraph and Telephone Corporation（日本电报电话公司）
This paper proposes a voice conversion (VC) method using sequence-to-sequence (seq2seq or S2S) learning, which flexibly converts not only the voice characteristics but also the pitch contour and duration of input speech. The proposed method, called ConvS2S-VC, has three key features. First, it uses a model with a fully convolutional architecture. This is particularly advantageous in that it is suitable for parallel computations using GPUs. It is also beneficial since it enables effective normalization techniques such as batch normalization to be used for all the hidden layers in the networks. …
We describe Parrotron, an end-to-end-trained speech-to-speech conversion model that maps an input spectrogram directly to another spectrogram, without utilizing any intermediate discrete representation. The network is composed of an encoder, spectrogram and phoneme decoders, followed by a vocoder to synthesize a time-domain waveform. We demonstrate that this model can be trained to normalize speech from any speaker regardless of accent, prosody, and background noise, into the voice of a single canonical target speaker with a fixed accent and consistent articulation and prosody. We further show…
Degrading Voice: A Comprehensive Overview of Robust Voice Conversion Through Input Manipulation
Tongji University（同济大学）；iFLYTEK Research（iFLYTEK研究院）；Binjiang Institute Of Zhejiang University（浙大滨江研究院）；Hefei University of Technology（合肥工业大学）；Zhejiang University（浙江大学）
The awareness for biased ASR datasets or models has increased notably in recent years. Even for English, despite a vast amount of available training data, systems perform worse for non-native speakers. In this work, we improve an accent-conversion model (ACM) which transforms native US-English speech into accented pronunciation. We include phonetic knowledge in the ACM training to provide accurate feedback about how well certain pronunciation patterns were recovered in the synthesized waveform. Furthermore, we investigate the feasibility of learned accent representations instead of static embe…
TokAN: Accent Normalization Using Self-Supervised Speech Tokens
School of Data Science (SDS), The Chinese University of Hong Kong, Shenzhen (CUHKSZ)（香港中文大学（深圳）数据科学学院）；Tencent Ethereal Audio Lab, Tencent（腾讯虚幻音频实验室）；School of Intelligence Science and Technology, Nanjing University, Suzhou（南京大学苏州校区智能科学与技术学院）；Shenzhen Loop Area Institute（深圳河套学院）；X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与工程学院X-LANCE实验室）；School of Artificial Intelligence (SAI), CUHKSZ（香港中文大学（深圳）人工智能学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Speech data conveys sensitive speaker attributes like identity or accent. With a small amount of found data, such attributes can be inferred and exploited for malicious purposes: voice cloning, spoofing, etc. Anonymization aims to make the data unlinkable, i.e., ensure that no utterance can be linked to its original speaker. In this paper, we investigate anonymization methods based on voice conversion. In contrast to prior work, we argue that various linkage attacks can be designed depending on the attackers' knowledge about the anonymization scheme. We compare two frequency warping-based conv…
The rapid spread of media content synthesis technology and the potentially damaging impact of audio and video deepfakes on people's lives have raised the need to implement systems able to detect these forgeries automatically. In this work we present a novel approach for synthetic speech detection, exploiting the combination of two high-level semantic properties of the human voice. On one side, we focus on speaker identity cues and represent them as speaker embeddings extracted using a state-of-the-art method for the automatic speaker verification task. On the other side, voice prosody, intende…
