Acoustic modeling for child speech is challenging due to the high acoustic variability caused by physiological differences in the vocal tract. The dearth of publicly available datasets makes the task more challenging. In this work, we propose a feature adaptation approach by exploiting adversarial multi-task training to minimize acoustic mismatch at the senone (tied triphone states) level between adult and child speech and leverage large amounts of transcribed adult speech. We validate the proposed method on three tasks: child speech recognition, child pronunciation assessment, and child fluency score prediction. Empirical results indicate that our proposed approach consistently outperforms competitive baselines, achieving 7.7% relative error reduction on speech recognition and up to 25.2% relative gains on the evaluation tasks.
The performance of child speech recognition is generally less satisfactory compared to adult speech due to limited amount of training data. Significant performance degradation is expected when applying an automatic speech recognition (ASR) system trained on adult speech to child speech directly, as a result of domain mismatch. The present study is focused on adult-to-child acoustic feature conversion to alleviate this mismatch. Different acoustic feature conversion approaches, including deep neural network based and signal processing based, are investigated and compared under a fair experiment…
Children's speech recognition is a vital, yet largely overlooked domain when building inclusive speech technologies. The major challenge impeding progress in this domain is the lack of adequate child speech corpora; however, recent advances in self-supervised learning have created a new opportunity for overcoming this problem of data scarcity. In this paper, we leverage self-supervised adult speech representations and use three well-known child speech corpora to build models for children's speech recognition. We assess the performance of fine-tuning on both native and non-native children's spe…
Evaluation of Speech Foundation Models for ASR on Child-Adult Conversations in Autism Diagnostic Sessions
Signal Analysis and Interpretation Laboratory（信号分析与解释实验室）；Department of Psychiatry（精神病学系）；Semel Institute of Neuroscience and Human Behavior（Semel神经科学与人类行为研究所）
Automatic Speech Recognition (ASR) systems are known to exhibit difficulties when transcribing children's speech. This can mainly be attributed to the absence of large children's speech corpora to train robust ASR models and the resulting domain mismatch when decoding children's speech with systems trained on adult data. In this paper, we propose multiple enhancements to alleviate these issues. First, we propose a data augmentation technique based on the source-filter model of speech to close the domain gap between adult and children's speech. This enables us to leverage the data availability …
Children speech recognition is challenging mainly due to the inherent high variability in children's physical and articulatory characteristics and expressions. This variability manifests in both acoustic constructs and linguistic usage due to the rapidly changing developmental stage in children's life. Part of the challenge is due to the lack of large amounts of available children speech data for efficient modeling. This work attempts to address the key challenges using transfer learning from adult's models to children's models in a Deep Neural Network (DNN) framework for children's Automatic …
End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions
Viterbi School of Engineering, University of Southern California, US（维特比工程学院，南加州大学）；Weill Institute for Neurosciences, University of California, San Francisco, US（韦尔神经科学研究所，旧金山大学）；David Geffen School of Medicine, University of California, Los Angeles, US（大卫·盖芬医学院，洛杉矶大学）
University of Trento（特伦托大学）；University of Groningen（格罗宁根大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会（FBK））
Pronunciation is one of the fundamentals of language learning, and it is considered a primary factor of spoken language when it comes to an understanding and being understood by others. The persistent presence of high error rates in speech recognition domains resulting from mispronunciations motivates us to find alternative techniques for handling mispronunciations. In this study, we develop a mispronunciation assessment system that checks the pronunciation of non-native English speakers, identifies the commonly mispronounced phonemes of Italian learners of English, and presents an evaluation …
Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息科技创新研究中心）
Children speech recognition is indispensable but challenging due to the diversity of children's speech. In this paper, we propose a filter-based discriminative autoencoder for acoustic modeling. To filter out the influence of various speaker types and pitches, auxiliary information of the speaker and pitch features is input into the encoder together with the acoustic features to generate phonetic embeddings. In the training phase, the decoder uses the auxiliary information and the phonetic embedding extracted by the encoder to reconstruct the input acoustic features. The autoencoder is trained…
University of Galway（戈尔韦大学）；University Politehnica of Bucharest（布加勒斯特理工大学）
Despite recent advancements in deep learning technologies, Child Speech Recognition remains a challenging task. Current Automatic Speech Recognition (ASR) models require substantial amounts of annotated data for training, which is scarce. In this work, we explore using the ASR model, wav2vec2, with different pretraining and finetuning configurations for self-supervised learning (SSL) toward improving automatic child speech recognition. The pretrained wav2vec2 models were finetuned using different amounts of child speech training data, adult speech data, and a combination of both, to discover t…
Technische Hochschule Nürnberg Georg Simon Ohm（纽伦堡乔治·西蒙·欧姆应用科学大学）；Intel Labs（英特尔实验室）
This work aims to automatically evaluate whether the language development of children is age-appropriate. Validated speech and language tests are used for this purpose to test the auditory memory. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures: Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR acoustic model). Each of the approaches provides input for VGG-like 5…
Paul Sabatier University（保罗萨巴蒂尔大学）；CNRS（法国国家科学研究中心）；Lalilo（拉利洛公司）
Automatic recognition systems for child speech are lagging behind those dedicated to adult speech in the race of performance. This phenomenon is due to the high acoustic and linguistic variability present in child speech caused by their body development, as well as the lack of available child speech data. Young readers speech additionally displays peculiarities, such as slow reading rate and presence of reading mistakes, that hardens the task. This work attempts to tackle the main challenges in phone acoustic modelling for young child speech with limited data, and improve understanding of stre…
A key desiderata for inclusive and accessible speech recognition technology is ensuring its robust performance to children's speech. Notably, this includes the rapidly advancing neural network based end-to-end speech recognition systems. Children speech recognition is more challenging due to the larger intra-inter speaker variability in terms of acoustic and linguistic characteristics compared to adult speech. Furthermore, the lack of adequate and appropriate children speech resources adds to the challenge of designing robust end-to-end neural architectures. This study provides a critical asse…
Automatic speech recognition (ASR) systems for young children are needed due to the importance of age-appropriate educational technology. Because of the lack of publicly available young child speech data, feature extraction strategies such as feature normalization and data augmentation must be considered to successfully train child ASR systems. This study proposes a novel technique for child ASR using both feature normalization and data augmentation methods based on the relationship between formants and fundamental frequency ($f_o$). Both the $f_o$ feature normalization and data augmentation t…
Automatic speech recognition (ASR) has been significantly advanced with the use of deep learning and big data. However improving robustness, including achieving equally good performance on diverse speakers and accents, is still a challenging problem. In particular, the performance of children speech recognition (CSR) still lags behind due to 1) the speech and language characteristics of children's voice are substantially different from those of adults and 2) sizable open dataset for children speech is still not available in the research community. To address these problems, we launch the Child…
University of Southern California（南加州大学）；Weill Cornell Medicine（威尔康奈尔医学院）；University of California Los Angeles（加州大学洛杉矶分校）
Computational modeling of naturalistic conversations in clinical applications has seen growing interest in the past decade. An important use-case involves child-adult interactions within the autism diagnosis and intervention domain. In this paper, we address a specific sub-problem of speaker diarization, namely child-adult speaker classification in such dyadic conversations with specified roles. Training a speaker classification system robust to speaker and channel conditions is challenging due to inherent variability in the speech within children and the adult interlocutors. In this work, we …
Egocentric Speaker Classification in Child-Adult Dyadic Interactions: From Sensing to Computational Modeling
University of Southern California（南加州大学）；University of California, San Francisco（加州大学旧金山分校）；Signal Analysis and Interpretation Laboratory（信号分析与解释实验室）；Weill Institute for Neurosciences（魏尔神经科学研究所）
ChildAugment: Data Augmentation Methods for Zero-Resource Children's Speaker Verification
School of Computing, University of Eastern Finland（东芬兰大学计算机学院）；Institute for Advancing Intelligence, TCG CREST（TCG CREST智能推进研究所）；Academy of Scientific and Innovative Research (AcSIR)（科学与创新研究院）
Speech Separation for Hearing-Impaired Children in the Classroom
Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition, and Behaviour（大脑、认知与行为研究所）；Leiden University Medical Centre（莱顿大学医学中心）；Mortimer B. Zuckerman Mind, Brain, Behavior Institute（莫蒂默·B·祖克erman心智、大脑与行为研究所）；Columbia University（哥伦比亚大学）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿大脑与认知研究所）；Delft University of Technology（代尔夫特理工大学）
University of Southern California（南加州大学）；University of California, San Francisco（加州大学旧金山分校）
Diagnostic procedures for ASD (autism spectrum disorder) involve semi-naturalistic interactions between the child and a clinician. Computational methods to analyze these sessions require an end-to-end speech and language processing pipeline that go from raw audio to clinically-meaningful behavioral features. An important component of this pipeline is the ability to automatically detect who is speaking when i.e., perform child-adult speaker classification. This binary classification task is often confounded due to variability associated with the participants' speech and background conditions. F…
Korean aegyo speech shows systematic F1 increase to signal childlike qualities
Department of Korean Language and Literature, Duksung Women’s University（韩国语言文学系，杜庆女子大学）；Department of Computational Linguistics, University of Zurich（计算语言学系，苏黎世大学）
Can large audio language models understand child stuttering speech? speech summarization, and source separation
Department of Computer Science, University of Florida, Gainesville, FL, USA（佛罗里达大学计算机科学系）；Department of Computer Science, Lebanese American University（黎巴嫩美国大学计算机科学系）
School of Electronic, Electrical and Computer Engineering, University of Birmingham（伯明翰大学电子、电气与计算机工程学院）
This paper presents results on Speaker Recognition (SR) for children's speech, using the OGI Kids corpus and GMM-UBM and GMM-SVM SR systems. Regions of the spectrum containing important speaker information for children are identified by conducting SR experiments over 21 frequency bands. As for adults, the spectrum can be split into four regions, with the first (containing primary vocal tract resonance information) and third (corresponding to high frequency speech sounds) being most useful for SR. However, the frequencies at which these regions occur are from 11% to 38% higher for children. It …
ChildGuard: A Specialized Dataset for Combatting Child-Targeted Hate Speech
Macquarie University（麦考瑞大学）；MBZUAI（穆罕默德·本·拉希德人工智能研究所）；DSEU（德里国家理工学院）；Department of Information and Computer Science, KFUPM（科威特石油大学信息与计算机科学系）；Stanford University（斯坦福大学）
