首页/论文/2001.10603

arXiv 2001.106032020-01eess.AScs.SD

Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction

Amazon Alexa（亚马逊Alexa）

Weiran Wang, Qingming Tang, Karen Livescu

备注：: Final version for ICASSP 2020

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

通过掩码语言模型校正离散自监督语音单元实现无监督口音自适应

Unsupervised Accent Adaptation Through Masked Language Model Correction Of Discrete Self-Supervised Speech Units

KU Leuven Department Electrical Engineering ESAT-PSI, Leuven, Belgium

本文提出一种无监督口音自适应方法，利用掩码语言模型校正离散语音单元，并通过小型适配器微调预训练模型，从而提升HuBERT Large在带口音语音识别上的鲁棒性。

2023-09-25cs.SDeess.AS

Joint Masked CPC and CTC Training for ASR

Facebook AI Research（脸书人工智能研究院）

Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR). But, training SSL models like wav2vec~2.0 requires a two-stage pipeline. In this paper we demonstrate a single-stage training of ASR models that can utilize both unlabeled and labeled data. During training, we alternately minimize two losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the supervised audio-to-text alignment loss Connectionist Temporal Classification (CTC). We show that this joint training method directly optimizes…

2021-02-13cs.CLcs.LGcs.SD

Ask2Mask: Guided Data Selection for Masked Speech Modeling

Brno University of Technology（布尔诺理工大学）

Masked speech modeling (MSM) methods such as wav2vec2 or w2v-BERT learn representations over speech frames which are randomly masked within an utterance. While these methods improve performance of Automatic Speech Recognition (ASR) systems, they have one major limitation. They treat all unsupervised speech samples with equal weight, which hinders learning as not all samples have relevant information to learn meaningful representations. In this work, we address this limitation. We propose ask2mask (ATM), a novel approach to focus on specific samples during MSM pre-training. ATM employs an exter…

2022-02-24cs.CLcs.SDeess.AS

A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition

Didi Chuxing（滴滴出行）

Building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, many unsupervised pre-training methods have been proposed. Among these methods, Masked Predictive Coding achieved significant improvements on various speech recognition datasets with BERT-like Masked Reconstruction loss and Transformer backbone. However, many aspects of MPC have not been fully investigated. In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pre-training data speaking style, its ext…

2020-06-23cs.CLcs.SDeess.AS

Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

AI Labs, Didi Chuxing（滴滴出行人工智能实验室）

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, w…

2019-10-31cs.CLcs.SDeess.AS

Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training

Microsoft（微软）

Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. It usually requires a codebook obtained in an unsupervised way, making it less accurate and difficult to interpret. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance and also the pre-training efficiency, either through decoding with a hybrid ASR system to generate phoneme-level alignments (named PBERT), or performing clustering on the supervised speech features extracted from an end-to-end CTC mo…

2022-06-21cs.CLcs.SDeess.AS

更多相关论文收起更多论文

Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders

National Taiwan University（台湾大学）

We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spok…

2020-02-02cs.CLcs.LGcs.SD

Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR

University of California Los Angeles（加州大学洛杉矶分校）

We present a bidirectional unsupervised model pre-training (UPT) method and apply it to children's automatic speech recognition (ASR). An obstacle to improving child ASR is the scarcity of child speech databases. A common approach to alleviate this problem is model pre-training using data from adult speech. Pre-training can be done using supervised (SPT) or unsupervised methods, depending on the availability of annotations. Typically, SPT performs better. In this paper, we focus on UPT to address the situations when pre-training data are unlabeled. Autoregressive predictive coding (APC), a UPT…

2021-02-12cs.LGeess.AS

使用生成对抗网络与人在回路评估指标的无监督佩戴口罩语音增强

Unsupervised Face-Masked Speech Enhancement Using Generative Adversarial Networks With Human-in-the-Loop Assessment Metrics

针对佩戴口罩导致的语音质量下降问题，提出HL-StarGAN无监督增强方法，引入含人类评估的MaskQSS模块，在FMVD数据库上优于现有方法。

2024-07-20eess.ASeess.SP

Masked Pre-trained Encoder base on Joint CTC-Transformer

This study (The work was accomplished during the internship in Tencent AI lab) addresses semi-supervised acoustic modeling, i.e. attaining high-level representations from unsupervised audio data and fine-tuning the parameters of pre-trained model with supervised data. The proposed approach adopts a two-stage training framework, consisting of masked pre-trained encoder (MPE) and Joint CTC-Transformer (JCT). In the MPE framework, part of input frames are masked and reconstructed after the encoder with massive unsupervised data. In JCT framework, compared with original Transformer, acoustic featu…

2020-08-13eess.AS