arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.01275eess.AScs.CLcs.LGcs.SD
收藏

放松注意力:一种提高端到端自动语音识别性能的简单方法

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Technische Universit\"at Braunschweig Institute for Communications Technology Schleinitzstr. 22, 38106 Braunschweig, Germany 12 12 \ , , , \

Timo Lohrenz, Patrick Schwarz, Zhengyang Li, Tim Fingscheidt

摘要:

最近,基于注意力的编码器-解码器(AED)模型已经在多个任务中显示出用于端到端自动语音识别(ASR)的高性能。为了解决这种模型中的过度自信,本文引入了放松注意力的概念,这是一个简单的逐步注入均匀分布的编码器-解码器注意力权重在训练过程中,很容易实现两行代码。我们研究了放松注意力在不同AED模型架构和两个突出的ASR任务,华尔街日报(WSJ)和Librispeech的影响。我们发现,在使用外部语言模型进行解码的过程中,放松注意力训练的Transformers始终优于标准基线模型。在《华尔街日报》上,我们为基于transformer的端到端语音识别设定了一个新的基准,单词错误率为3.65%,比最先进的水平(4.20%)高出13.1%,同时只引入了一个超参数。

英文摘要:

Recently, attention-based encoder-decoder (AED) models have shown high performance for end-to-end automatic speech recognition (ASR) across several tasks. Addressing overconfidence in such models, in this paper we introduce the concept of relaxed attention, which is a simple gradual injection of a uniform distribution to the encoder-decoder attention weights during training that is easily implemented with two lines of code. We investigate the effect of relaxed attention across different AED model architectures and two prominent ASR tasks, Wall Street Journal (WSJ) and Librispeech. We found that transformers trained with relaxed attention outperform the standard baseline models consistently during decoding with external language models. On WSJ, we set a new benchmark for transformer-based end-to-end speech recognition with a word error rate of 3.65%, outperforming state of the art (4.20%) by 13.1% relative, while introducing only a single hyperparameter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑