Whispering is an important mode of human speech, but no end-to-end recognition results for it were reported yet, probably due to the scarcity of available whispered speech data. In this paper, we present several approaches for end-to-end (E2E) recognition of whispered speech considering the special characteristics of whispered speech and the scarcity of data. This includes a frequency-weighted SpecAugment policy and a frequency-divided CNN feature extractor for better capturing the high-frequency structures of whispered speech, and a layer-wise transfer learning approach to pre-train a model with normal or normal-to-whispered converted speech then fine-tune it with whispered speech to bridge the gap between whispered and normal speech. We achieve an overall relative reduction of 19.8% in PER and 44.4% in CER on a relatively small whispered TIMIT corpus. The results indicate as long as we have a good E2E model pre-trained on normal or pseudo-whispered speech, a relatively small set of whispered speech may suffice to obtain a reasonably good E2E whispered speech recognizer.
State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition
Department of Computational Linguistics（计算语言学系）；Recognition Technologies（语音识别技术）；Faculty of New Sciences and Technologies（新科学与技术学院）；Faculty of New Sciences（新科学学院）
The University of Tokyo（东京大学）；Sony CSL Kyoto（索尼计算机科学实验室京都）
Interactions based on automatic speech recognition (ASR) have become widely used, with speech input being increasingly utilized to create documents. However, as there is no easy way to distinguish between commands being issued and text required to be input in speech, misrecognitions are difficult to identify and correct, meaning that documents need to be manually edited and corrected. The input of symbols and commands is also challenging because these may be misrecognized as text letters. To address these problems, this study proposes a speech interaction method called DualVoice, by which comm…
WhisperMask: A Noise Suppressive Mask-Type Microphone for Whisper Speech
The University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology（独立行政法人产业技术综合研究所）；Sony Computer Science Laboratory（索尼计算机科学实验室）
We present an approach to synthesize whisper by applying a handcrafted signal processing recipe and Voice Conversion (VC) techniques to convert normally phonated speech to whispered speech. We investigate using Gaussian Mixture Models (GMM) and Deep Neural Networks (DNN) to model the mapping between acoustic features of normal speech and those of whispered speech. We evaluate naturalness and speaker similarity of the converted whisper on an internal corpus and on the publicly available wTIMIT corpus. We show that applying VC techniques is significantly better than using rule-based signal proce…
Machine recognition of an atypical speech like whispered speech, is a challenging task. We introduce whisper-to-natural-speech conversion using sequence-to-sequence approach by proposing enhanced transformer architecture, which uses both parallel and non-parallel data. We investigate different features like Mel frequency cepstral coefficients and smoothed spectral features. The proposed networks are trained end-to-end using supervised approach for feature-to-feature transformation. Further, we also investigate the effectiveness of embedded auxillary decoder used after N encoder sub-layers, tra…
It is desirable for a text-to-speech system to take into account the environment where synthetic speech is presented, and provide appropriate context-dependent output to the user. In this paper, we present and compare various approaches for generating different speaking styles, namely, normal, Lombard, and whisper speech, using only limited data. The following systems are proposed and assessed: 1) Pre-training and fine-tuning a model for each style. 2) Lombard and whisper speech conversion through a signal processing based approach. 3) Multi-style generation using a single model based on a spe…
CAFE A Novel Code switching Dataset for Algerian Dialect French and English
University of Bouira（布维拉大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Qatar Computing and Research Institute (HBKU)（卡塔尔计算与研究所（HBKU））
GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement
X-LANCE Lab, SCS, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（X-LANCE实验室、SCS、人工智能MoE重点实验室、上海交通大学）；Pengcheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；Dept EE, Tsinghua University（清华大学电子工程系）；SII；Harbin Institute of Technology（哈尔滨工业大学）；Birch AI；Dataocean AI；AISpeech Ltd（AISpeech有限公司）；Seasalt AI Inc（Seasalt AI公司）；SpeechColab
