Prosody is a rich information source in natural language, serving as a marker for phenomena such as contrast. In order to make this information available to downstream tasks, we need a way to detect prosodic events in speech. We propose a new model for pitch accent detection, inspired by the work of Stehwien et al. (2018), who presented a CNN-based model for this task. Our model makes greater use of context by using full utterances as input and adding an LSTM layer. We find that these innovations lead to an improvement from 87.5% to 88.7% accuracy on pitch accent detection on American English speech in the Boston University Radio News Corpus, a state-of-the-art result. We also find that a simple baseline that just predicts a pitch accent on every content word yields 82.2% accuracy, and we suggest that this is the appropriate baseline for this task. Finally, we conduct ablation tests that show pitch is the most important acoustic feature for this task and this corpus.
End-to-end text-to-speech synthesis (TTS) can generate highly natural synthetic speech from raw text. However, rendering the correct pitch accents is still a challenging problem for end-to-end TTS. To tackle the challenge of rendering correct pitch accent in Japanese end-to-end TTS, we adopt PnG~BERT, a self-supervised pretrained model in the character and phoneme domain for TTS. We investigate the effects of features captured by PnG~BERT on Japanese TTS by modifying the fine-tuning condition to determine the conditions helpful inferring pitch accents. We manipulate content of PnG~BERT feature…
Modern Automatic Speech Recognition (ASR) technology has evolved to identify the speech spoken by native speakers of a language very well. However, identification of the speech spoken by non-native speakers continues to be a major challenge for it. In this work, we first spell out the key requirements for creating a well-curated database of speech samples in non-native accents for training and testing robust ASR systems. We then introduce AccentDB, one such database that contains samples of 4 Indian-English accents collected by us, and a compilation of samples from 4 native-English, and a metr…
By strictest interpretation, theories of both centering and intonational meaning fail to predict the existence of pitch accented pronominals. Yet they occur felicitously in spoken discourse. To explain this, I emphasize the dual functions served by pitch accents, as markers of both propositional (semantic/pragmatic) and attentional salience. This distinction underlies my proposals about the attentional consequences of pitch accents when applied to pronominals, in particular, that while most pitch accents may weaken or reinforce a cospecifier's status as the center of attention, a contrastively…
Speech to text models tend to be trained and evaluated against a single target accent. This is especially true for English for which native speakers from the United States became the main benchmark. In this work, we are going to show how two simple methods: pre-trained embeddings and auxiliary classification losses can improve the performance of ASR systems. We are looking for upgrades as universal as possible and therefore we will explore their impact on several models architectures and several languages.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
The performance of automatic speech recognition systems degrades with increasing mismatch between the training and testing scenarios. Differences in speaker accents are a significant source of such mismatch. The traditional approach to deal with multiple accents involves pooling data from several accents during training and building a single model in multi-task fashion, where tasks correspond to individual accents. In this paper, we explore an alternate model where we jointly learn an accent classifier and a multi-task acoustic model. Experiments on the American English Wall Street Journal and…
Indian Institute of Science Education and Research (IISER) Berhampur（贝兰布尔印度科学教育与研究院（IISER））
Automated Speech Recognition (ASR) is an interdisciplinary application of computer science and linguistics that enable us to derive the transcription from the uttered speech waveform. It finds several applications in Military like High-performance fighter aircraft, helicopters, air-traffic controller. Other than military speech recognition is used in healthcare, persons with disabilities and many more. ASR has been an active research area. Several models and algorithms for speech to text (STT) have been proposed. One of the most recent is Mozilla Deep Speech, it is based on the Deep Speech res…
Deep Supervised Contrastive Learning of Pitch Contours for Robust Pitch Accent Classification in Seoul Korean
Department of Linguistics, Rutgers University（鲁特大学语言学系）；Department of Smart Factory, Gachon University（加荣大学智能工厂系）；Hanyang Institute for Phonetics and Cognitive Sciences of Language (HIPCS)（翰阳语言语音与认知科学研究所）
National Institute of Informatics（国立信息学研究所）；The University of Edinburgh（爱丁堡大学）；SOKENDAI (The Graduate University for Advanced Studies)（综合研究大学院大学（高等研究生院大学））
End-to-end speech synthesis is a promising approach that directly converts raw text to speech. Although it was shown that Tacotron2 outperforms classical pipeline systems with regards to naturalness in English, its applicability to other languages is still unknown. Japanese could be one of the most difficult languages for which to achieve end-to-end speech synthesis, largely due to its character diversity and pitch accents. Therefore, state-of-the-art systems are still based on a traditional pipeline framework that requires a separate text analyzer and duration model. Towards end-to-end Japane…
Pitch accent detection often makes use of both acoustic and lexical features based on the fact that pitch accents tend to correlate with certain words. In this paper, we extend a pitch accent detector that involves a convolutional neural network to include word embeddings, which are state-of-the-art vector representations of words. We examine the effect these features have on within-corpus and cross-corpus experiments on three English datasets. The results show that while word embeddings can improve the performance in corpus-dependent experiments, they also have the potential to make generaliz…
Probing for Phonology in Self-Supervised Speech Representations: A Case Study on Accent Perception
Department of Linguistics, University of Florida（语言学系，佛罗里达大学）；Department of English Language and Linguistics, Heinrich Heine University Düsseldorf（英语语言与语言学系，海因里希-海涅大学杜塞尔多夫）
The awareness for biased ASR datasets or models has increased notably in recent years. Even for English, despite a vast amount of available training data, systems perform worse for non-native speakers. In this work, we improve an accent-conversion model (ACM) which transforms native US-English speech into accented pronunciation. We include phonetic knowledge in the ACM training to provide accurate feedback about how well certain pronunciation patterns were recovered in the synthesized waveform. Furthermore, we investigate the feasibility of learned accent representations instead of static embe…
Universitat Pompeu Fabra（庞培法布拉大学）；Telefónica Research（西班牙电信研究院）；Universitat de Barcelona（巴塞罗那大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Nowadays, research in speech technologies has gotten a lot out thanks to recently created public domain corpora that contain thousands of recording hours. These large amounts of data are very helpful for training the new complex models based on deep learning technologies. However, the lack of dialectal diversity in a corpus is known to cause performance biases in speech systems, mainly for underrepresented dialects. In this work, we propose to evaluate a state-of-the-art automatic speech recognition (ASR) deep learning-based model, using unseen data from a corpus with a wide variety of labeled…
National Institute of Informatics（国立信息学研究所）；KDDI Research Inc.（KDDI研究所）
We investigated the impact of noisy linguistic features on the performance of a Japanese speech synthesis system based on neural network that uses WaveNet vocoder. We compared an ideal system that uses manually corrected linguistic features including phoneme and prosodic information in training and test sets against a few other systems that use corrupted linguistic features. Both subjective and objective results demonstrate that corrupted linguistic features, especially those in the test set, affected the ideal system's performance significantly in a statistical sense due to a mismatched condi…
AI Saturdays Lagos（拉各斯AI星期六）；Niger-Volta Language Technologies Institute（尼日尔-沃尔特语言技术研究所）
This paper describes foundational efforts with SautiDB-Naija, a novel corpus of non-native (L2) Nigerian English speech. We describe how the corpus was created and curated as well as preliminary experiments with accent classification and learning Nigerian accent embeddings. The initial version of the corpus includes over 900 recordings from L2 English speakers of Nigerian languages, such as Yoruba, Igbo, Edo, Efik-Ibibio, and Igala. We further demonstrate how fine-tuning on a pre-trained model like wav2vec can yield representations suitable for related speech tasks such as accent classificatio…
Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking
Department of Computer Engineering, Sharif University of Technology（谢里夫理工大学计算机工程系）；Department of Computer Engineering, University of Tehran（德黑兰大学计算机工程系）
Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data
Signal Analysis and Interpretation Lab, University of Southern California, USA（信号分析与解释实验室，南加州大学）；Thomas Lord Department of Computer Science, University of Southern California, USA（托马斯·劳德计算机科学系，南加州大学）；Department of Linguistics, University of Southern California（语言学系，南加州大学）
Inner Mongolia University（内蒙古大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Bremen（不来梅大学）
Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1). Accented TTS synthesis is challenging as L2 is different from L1 in both in terms of phonetic rendering and prosody pattern. Furthermore, there is no easy solution to the control of the accent intensity in an utterance. In this work, we propose a neural TTS architecture, that allows us to control the accent and its intensity during inference. This is achieved through three novel mechanisms, 1) an accent variance adaptor to model the complex accent variance with three …
This paper describes our experiments with automatically identifying native accents from speech samples of non-native English speakers using low level audio features, and n-gram features from manual transcriptions. Using a publicly available non-native speech corpus and simple audio feature representations that do not perform word/phoneme recognition, we show that it is possible to achieve close to 90% classification accuracy for this task. While character n-grams perform similar to speech features, we show that speech features are not affected by prompt variation, whereas ngrams are. Since the…
As one of the major sources in speech variability, accents have posed a grand challenge to the robustness of speech recognition systems. In this paper, our goal is to build a unified end-to-end speech recognition system that generalizes well across accents. For this purpose, we propose a novel pre-training framework AIPNet based on generative adversarial nets (GAN) for accent-invariant representation learning: Accent Invariant Pre-training Networks. We pre-train AIPNet to disentangle accent-invariant and accent-specific characteristics from acoustic features through adversarial training on acc…
Speech systems are sensitive to accent variations. This is especially challenging in the Indian context, with an abundance of languages but a dearth of linguistic studies characterising pronunciation variations. The growing number of L2 English speakers in India reinforces the need to study accents and L1-L2 interactions. We investigate the accents of Indian English (IE) speakers and report in detail our observations, both specific and common to all regions. In particular, we observe the phonemic variations and phonotactics occurring in the speakers' native languages and apply this to their En…
Previous accent classification research focused mainly on detecting accents with pure acoustic information without recognizing accented speech. This work combines phonetic knowledge such as vowels with acoustic information to build Guassian Mixture Model (GMM) classifier with Perceptual Linear Predictive (PLP) features, optimized by Hetroscedastic Linear Discriminant Analysis (HLDA). With input about 20-second accented speech, this system achieves classification rate of 51% on a 7-way classification system focusing on the major types of accents in English, which is competitive to the state-of-…
To address the performance gap of English ASR models on L2 English speakers, we evaluate fine-tuning of pretrained wav2vec 2.0 models (Baevski et al., 2020; Xu et al., 2021) on L2-ARCTIC, a non-native English speech corpus (Zhao et al., 2018) under different training settings. We compare \textbf{(a)} models trained with a combination of diverse accents to ones trained with only specific accents and \textbf{(b)} results from different single-accent models. Our experiments demonstrate the promise of developing ASR models for non-native English speakers, even with small amounts of L2 training dat…
Past research has identified discriminatory automatic speech recognition (ASR) performance as a function of the racial group and nationality of the speaker. In this paper, we expand the discussion beyond bias as a function of the individual national origin of the speaker to look for bias as a function of the geopolitical orientation of their nation of origin. We audit some of the most popular English language ASR services using a large and global data set of speech from The Speech Accent Archive, which includes over 2,700 speakers of English born in 171 different countries. We show that, even …
Accented speech recognition and accent classification are relatively under-explored research areas in speech technology. Recently, deep learning-based methods and Transformer-based pretrained models have achieved superb performances in both areas. However, most accent classification tasks focused on classifying different kinds of English accents and little attention was paid to geographically-proximate accent classification, especially under a low-resource setting where forensic speech science tasks usually encounter. In this paper, we explored three main accent modelling methods combined with…
Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；LORIA（洛林计算机科学与应用实验室）；Intron Health（Intron健康公司）；Amazethu Research（Amazethu研究所）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；AI Saturdays Lagos（拉各斯AI周六组织）；University of Pretoria（比勒陀利亚大学）；University of Minnesota - Twin Cities（明尼苏达大学双城分校）；Luleå University of Technology（吕勒奥理工大学）；Masakhane NLP（马萨卡内自然语言处理组织）
Carnegie Mellon University（卡内基梅隆大学）；Indian Institute of Technology Bombay（印度理工学院孟买分校）；Indian Institute of Science（印度科学学院）
The problem of automatic accent identification is important for several applications like speaker profiling and recognition as well as for improving speech recognition systems. The accented nature of speech can be primarily attributed to the influence of the speaker's native language on the given speech recording. In this paper, we propose a novel accent identification system whose training exploits speech in native languages along with the accented speech. Specifically, we develop a deep Siamese network-based model which learns the association between accented speech recordings and the native…
Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis
Signal Analysis and Interpretation Lab, University of Southern California（南加州大学信号分析与解释实验室）；Thomas Lord Department of Computer Science, University of Southern California（南加州大学托马斯·劳德计算机科学系）；Department of Linguistics, University of Southern California（南加州大学语言学系）
本研究通过量化说话者嵌入与语音规则的相互作用，提出 PSR 度量标准，揭示口音生成中规则与身份的纠缠关系。
Connecting the Dots: Enduring Performance Gaps in ASR Systems Between Standard and Minority English Varieties
Koel Labs LLC（科尔实验室有限责任公司）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Toronto（多伦多大学）；University of Essex（埃塞克斯大学）；The Ohio State University（俄亥俄州立大学）
Institute for Natural Language Processing（自然语言处理研究所）；University of Stuttgart（斯图加特大学）
Adding manually annotated prosodic information, specifically pitch accents and phrasing, to the typical text-based feature set for coreference resolution has previously been shown to have a positive effect on German data. Practical applications on spoken language, however, would rely on automatically predicted prosodic information. In this paper we predict pitch accents (and phrase boundaries) using a convolutional neural network (CNN) model from acoustic features extracted from the speech signal. After an assessment of the quality of these automatic prosodic annotations, we show that they als…
The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs
1 Department of Speech, Music；Hearing, KTH Royal Institute of Technology, Sweden 2 Centre for Speech Technology Research, University of Edinburgh, UK 3 Texas A\&M University, USA
Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond
Intron（英创公司）；BioRAMP（BioRAMP机构）；Brown University（布朗大学）；Indian Institute of Information Technology Allahabad（阿拉哈巴德印度信息技术学院）；University of Minnesota-Twin Cities（明尼苏达大学双城分校）；University of Glasgow（格拉斯哥大学）；University of Florida（佛罗里达大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Georgia Institute of Technology（佐治亚理工学院）
该工作提出 Afrispeech-Dialog 基准数据集，通过 50 段非洲口音英语对话评估 ASR 和说话人分离系统，发现口音导致性能下降超 10%，并揭示 ASR 错误对医疗摘要的影响，呼吁构建更具包容性的低资源语音数据集。
CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice
Idiap Research Institute（Idiap研究所）；Ecole Polytechnique Federale de Lausanne（洛桑联邦理工学院）；Texas A&M University（得克萨斯农工大学）；Vilnius Gediminas Technical University（维尔纽斯盖迪米纳斯技术大学）；Université Laval（拉瓦尔大学）；Concordia University（康考迪亚大学）；Mila-Québec AI Institute（米拉-魁北克人工智能研究所）
Learning-free L2-Accented Speech Generation using Phonological Rules
Signal Analysis and Interpretation Lab, University of Southern California, USA（信号分析与解读实验室，美国南加州大学）；Department of Linguistics, University of Southern California（语言学系，美国南加州大学）
Modern automatic speech recognition (ASR) systems have achieved superhuman Word Error Rate (WER) on many common corpora despite lacking adequate performance on speech in the wild. Beyond that, there is a lack of real-world, accented corpora to properly benchmark academic and commercial models. To ensure this type of speech is represented in ASR benchmarking, we present Earnings-22, a 125 file, 119 hour corpus of English-language earnings calls gathered from global companies. We run a comparison across 4 commercial models showing the variation in performance when taking country of origin into c…
Universitat Pompeu Fabra（庞培法布拉大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
The goal of this study is the automatic detection of onsets of the singing voice in polyphonic audio recordings. Starting with a hypothesis that the knowledge of the current position in a metrical cycle (i.e. metrical accent) can improve the accuracy of vocal note onset detection, we propose a novel probabilistic model to jointly track beats and vocal note onsets. The proposed model extends a state of the art model for beat and meter tracking, in which a-priori probability of a note at a specific metrical accent interacts with the probability of observing a vocal note onset. We carry out an ev…
Brno University of Technology（布尔诺理工大学）；Sharif DeepMine Ltd.（谢里夫DeepMine有限公司）
DeepMine is a speech database in Persian and English designed to build and evaluate text-dependent, text-prompted, and text-independent speaker verification, as well as Persian speech recognition systems. It contains more than 1850 speakers and 540 thousand recordings overall, more than 480 hours of speech are transcribed. It is the first public large-scale speaker verification database in Persian, the largest public text-dependent and text-prompted speaker verification database in English, and the largest public evaluation dataset for text-independent speaker verification. It has a good cover…
