Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis results with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent results and we recommend their use as a complement to local-scope diagnostic methods.
In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation and intonation of the second language are usually quite different due to the influence of the first language. Therefore, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingua…
We study the representation and encoding of phonemes in a recurrent neural network model of grounded speech. We use a model which processes images and their spoken descriptions, and projects the visual and auditory representations into the same semantic space. We perform a number of analyses on how information about individual phonemes is encoded in the MFCC features extracted from the speech signal, and the activations of the layers of the model. Via experiments with phoneme decoding and phoneme discrimination we show that phoneme representations are most salient in the lower layers of the mo…
School of Informatics（信息学院）；University of Edinburgh（爱丁堡大学）；E&E Engineering（电气与电子工程学院）；Stellenbosch University（斯泰伦博斯大学）；University of Maryland（马里兰大学）
Non-native speakers show difficulties with spoken word processing. Many studies attribute these difficulties to imprecise phonological encoding of words in the lexical memory. We test an alternative hypothesis: that some of these difficulties can arise from the non-native speakers' phonetic perception. We train a computational model of phonetic learning, which has no access to phonology, on either one or two languages. We first show that the model exhibits predictable behaviors on phone-level and word-level discrimination tasks. We then test the model on a spoken word processing task, showing …
Department of Computer Science（计算机科学系）；University of Copenhagen（哥本哈根大学）；Denmark（丹麦）
A core part of linguistic typology is the classification of languages according to linguistic properties, such as those detailed in the World Atlas of Language Structure (WALS). Doing this manually is prohibitively time-consuming, which is in part evidenced by the fact that only 100 out of over 7,000 languages spoken in the world are fully covered in WALS. We learn distributed language representations, which can be used to predict typological properties on a massively multilingual scale. Additionally, quantitative and qualitative analyses of these language embeddings can tell us how language s…
Emergent morpho-phonological representations in self-supervised speech models
Department of Neurological Surgery, University of California, San Francisco（加州大学旧金山分校神经外科系）；Department of Linguistics, University of Southern California（南加州大学语言学系）；Center for Computational Language Sciences, University of Southern California（南加州大学计算语言科学中心）
Introduction: Alzheimer's disease is a type of dementia in which early diagnosis plays a major rule in the quality of treatment. Among new works in the diagnosis of Alzheimer's disease, there are many of them analyzing the voice stream acoustically, syntactically or both. The mostly used tools to perform these analysis usually include machine learning techniques. Objective: Designing an automatic machine learning based diagnosis system will help in the procedure of early detection. Also, systems, using noninvasive data are preferable. Methods: We used are classification system based on spoken …
Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments. In this work, we study several pre-trained models and pooling methods for constructing AWEs with self-supervised representations. Owing to the contextualized nature of self-supervised representations, we hypothesize that simple pooling methods, such as averaging, might already be useful for constructing AWEs. When evaluating on a sta…
MoRoVoc: A Large Dataset for Geographical Variation Identification of the Spoken Romanian Language
National University of Science and Technology POLITEHNICA Bucharest（波兰亚技术大学布加勒斯特分校）；Paris 1 Panthéon-Sorbonne University（巴黎第一大学）；University of Bucharest（布加勒斯特大学）
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Karlsruhe Institute of Technology, Germany（卡尔斯鲁厄理工学院，德国）；Fondazione Bruno Kessler, Italy（布鲁诺·凯瑟尔基金会，意大利）；ACC Cyfronet AGH, Poland（AGH计算机科学与技术学院，波兰）；AGH University of Krakow, Poland（克拉科夫AGH大学，波兰）；Carnegie Mellon University, U.S.（卡内基梅隆大学，美国）
In this paper, we describe a so-called screening approach for learning robust processing of spontaneously spoken language. A screening approach is a flat analysis which uses shallow sequences of category representations for analyzing an utterance at various syntactic, semantic and dialog levels. Rather than using a deeply structured symbolic analysis, we use a flat connectionist analysis. This screening approach aims at supporting speech and language processing by using (1) data-driven learning and (2) robustness of connectionist networks. In order to test this approach, we have developed the …
In daily communications, Arabs use local dialects which are hard to identify automatically using conventional classification methods. The dialect identification challenging task becomes more complicated when dealing with an under-resourced dialects belonging to a same county/region. In this paper, we start by analyzing statistically Algerian dialects in order to capture their specificities related to prosody information which are extracted at utterance level after a coarse-grained consonant/vowel segmentation. According to these analysis findings, we propose a Hierarchical classification appro…
LIMSI（法国语言与信息系统实验室）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；LPP（巴黎语言实验室）；ENS（巴黎高等师范学院）；EHESS（法国高等社会科学研究院）；PSL Research University（巴黎文理研究大学）；INRIA（法国国家信息与自动化研究所）；LIG（格勒诺布尔信息学实验室）；UGA（格勒诺布尔阿尔卑斯大学）；G-INP（格勒诺布尔国立理工学院）；KIT（卡尔斯鲁厄理工学院）
Most speech and language technologies are trained with massive amounts of speech and text information. However, most of the world languages do not have such resources or stable orthography. Systems constructed under these almost zero resource conditions are not only promising for speech technology but also for computational language documentation. The goal of computational language documentation is to help field linguists to (semi-)automatically analyze and annotate audio recordings of endangered and unwritten languages. Example tasks are automatic phoneme discovery or lexicon discovery from t…
How Do Lexical Senses Correspond Between Spoken German and German Sign Language?
Institute for Computational Linguistics, University of Heidelberg（计算语言学研究所，海德堡大学）；Department of Computing Science, University of Aberdeen（计算机科学系，阿伯丁大学）
From Statistical Methods to Pre-Trained Models; A Survey on Automatic Speech Recognition for Resource Scarce Urdu Language
Shenzhen University（深圳大学）；School of Artificial Intelligence, CAS Institute of Automation（中国科学院自动化研究所人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）
University of Tartu（塔尔图大学）；Singidunum University（辛吉杜努姆大学）；University of Barcelona（巴塞罗那大学）；Hasan Kalyoncu University（哈桑·卡利翁库大学）
Every speech signal carries implicit information about the emotions, which can be extracted by speech processing methods. In this paper, we propose an algorithm for extracting features that are independent from the spoken language and the classification method to have comparatively good recognition performance on different languages independent from the employed classification methods. The proposed algorithm is composed of three stages. In the first stage, we propose a feature ranking method analyzing the state-of-the-art voice quality features. In the second stage, we propose a method for fin…
Sikkim Manipal Institute of Technology（锡金马尼帕尔理工学院）；Tezpur University（特兹普尔大学）
A natural language (or ordinary language) is a language that is spoken, written, or signed by humans for general-purpose communication, as distinguished from formal languages (such as computer-programming languages or the "languages" used in the study of formal logic). The computational activities required for enabling a computer to carry out information processing using natural language is called natural language processing. We have taken Assamese language to check the grammars of the input sentence. Our aim is to produce a technique to check the grammatical structures of the sentences in Ass…
