arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1802.06424cs.CV
收藏

端到端视听语音识别

End-to-end Audiovisual Speech Recognition

Imperial College London(帝国理工学院);University of Nottingham(诺丁汉大学)

Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Georgios Tzimiropoulos, Maja Pantic

摘要:

最近已经提出了几种端到端深度学习方法,其从输入图像或音频信号中提取音频或视觉特征并执行语音识别。然而,对端到端视听模型的研究非常有限。在这项工作中,我们提出了一个端到端的视听模型的基础上残留的网络和双向门控递归单元(BGRU)。据我们所知,这是第一个视听融合模型,它同时学习直接从图像像素和音频波形中提取特征,并在大型公开数据集(LRW)上执行上下文内单词识别。该模型由两个流组成,每个流用于一种模态,其直接从嘴部区域和原始波形提取特征。每个流/模态中的时间动态由2层BGRU建模,并且多个流/模态的融合经由另一个2层BGRU发生。在干净的音频条件和低噪声水平下,报告了端到端仅音频和基于MFCC的模型的分类率的轻微改善。在存在高水平噪声的情况下,端到端视听模型明显优于两种仅音频模型。

英文摘要:

Several end-to-end deep learning approaches have been recently presented which extract either audio or visual features from the input images or audio signals and perform speech recognition. However, research on end-to-end audiovisual models is very limited. In this work, we present an end-to-end audiovisual model based on residual networks and Bidirectional Gated Recurrent Units (BGRUs). To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the image pixels and audio waveforms and performs within-context word recognition on a large publicly available dataset (LRW). The model consists of two streams, one for each modality, which extract features directly from mouth regions and raw waveforms. The temporal dynamics in each stream/modality are modeled by a 2-layer BGRU and the fusion of multiple streams/modalities takes place via another 2-layer BGRU. A slight improvement in the classification rate over an end-to-end audio-only and MFCC-based model is reported in clean audio conditions and low levels of noise. In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑