arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02099eess.AScs.SD
收藏

LMEC:一种可学习的基于乘法绝对位置嵌入的语音识别保形器

LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition

Ximalaya Inc.(喜马拉雅公司);University of Alberta(阿尔伯塔大学)

Yuguang Yang, Yu Pan, Jingjing Yin, Heng Lu

摘要:

本文提出了一种可学习的基于乘法绝对位置嵌入的保形器(LMEC)。它包含一个称为LMLA的内核化线性注意力(LA)模块,以解决长序列语音识别的耗时问题,以及FFN结构的替代方案。首先,采用ELU函数作为我们提出的LA模块的核函数。其次,我们提出了一种新的基于可学习乘法绝对位置嵌入(LM-APE)的重新加权机制,可以降低softmax自注意的二次时空复杂度。第三,我们使用门控线性单元(GLU)来代替前馈网络(FFN),以获得更好的性能。已经在公共LibriSpeech数据集上进行了广泛的实验。与具有cosFormer风格线性注意力的Conformer模型相比,我们提出的方法可以在测试其他上实现高达0.63%的单词错误率改善,并在LA模块上将推理速度提高高达13%(左产品)和33%(右产品)。

英文摘要:

This paper proposes a Learnable Multiplicative absolute position Embedding based Conformer (LMEC). It contains a kernelized linear attention (LA) module called LMLA to solve the time-consuming problem for long sequence speech recognition as well as an alternative to the FFN structure. First, the ELU function is adopted as the kernel function of our proposed LA module. Second, we propose a novel Learnable Multiplicative Absolute Position Embedding (LM-APE) based re-weighting mechanism that can reduce the well-known quadratic temporal-space complexity of softmax self-attention. Third, we use Gated Linear Units (GLU) to substitute the Feed Forward Network (FFN) for better performance. Extensive experiments have been conducted on the public LibriSpeech datasets. Compared to the Conformer model with cosFormer style linear attention, our proposed method can achieve up to 0.63% word-error-rate improvement on test-other and improve the inference speed by up to 13% (left product) and 33% (right product) on the LA module.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑