arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.06028eess.AScs.CLcs.LG
收藏

TERA:语音Transformer编码器表示的自监督学习

TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

National Taiwan University(国立台湾大学);Amazon AI(亚马逊人工智能)

Andy T. Liu, Shang-Wen Li, Hung-yi Lee

摘要:

我们介绍了一种自监督的语音预训练方法,称为TERA,它代表来自Alteration的Transformer Encoder Representations。最近的方法通常通过使用单个辅助任务来学习,如对比预测,自回归预测或掩码重建。与以前的方法不同,我们使用沿三个正交轴的改变来在大量未标记的语音上预训练Transformer编码器。该模型通过从改变的对应物中重建声学帧来学习,在那里我们使用随机策略来改变各个维度:时间,频率和幅度。TERA可用于语音表示提取或与下游模型进行微调。我们评估TERA的几个下游任务,包括音素分类,关键字定位,说话人识别,和语音识别。我们对各种自监督模型进行了大规模的比较。TERA通过改进表面特征并优于以前的模型,在比较中实现了强大的性能。在我们的实验中,我们研究了应用不同的更改技术,对更多数据进行预训练以及对各种特征进行预训练的效果。我们分析了不同的模型大小,发现较小的模型比较大的模型更强的表示学习器,而较大的模型比较小的模型更有效地进行下游微调。此外,我们还证明了所提出的方法可以转移到预训练中未使用的下游数据集。

英文摘要:

We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑