arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1709.04343cs.CV
收藏

使用LSTM实现端到端视听融合

End-to-End Audiovisual Fusion with LSTMs

Imperial College London(帝国理工学院);University of Twente(特文特大学)

Stavros Petridis, Yujiang Wang, Zuwei Li, Maja Pantic

摘要:

最近已经提出了几种端到端的深度学习方法,其同时从输入图像中提取视觉特征并执行视觉语音分类。然而,联合提取音频和视觉特征并进行分类的研究非常有限。本文提出了一种基于双向长短期记忆(BLSTM)网络的端到端视听模型。据我们所知,这是第一个视听融合模型,它同时学习直接从像素和频谱图中提取特征,并对语音和非语言发声进行分类。该模型由多个相同的流组成,每个模态一个,直接从嘴部区域和频谱图中提取特征。每个流/模态中的时间动态由BLSTM建模,并且多个流/模态的融合经由另一BLSTM发生。在AVIC数据库中,4种非语言发声的平均F1比仅音频分类的绝对改善1.9%。与此同时,所提出的端到端视听融合系统提高了AVIC数据库的最新性能,使平均F1测量值绝对增加了9.7%。我们还进行视听语音识别实验的OuluVS 2数据库使用不同的意见,口,正面的配置文件。建议的视听系统显着优于音频只有模型的所有意见时,声学噪声高。

英文摘要:

Several end-to-end deep learning approaches have been recently presented which simultaneously extract visual features from the input images and perform visual speech classification. However, research on jointly extracting audio and visual features and performing classification is very limited. In this work, we present an end-to-end audiovisual model based on Bidirectional Long Short-Term Memory (BLSTM) networks. To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the pixels and spectrograms and perform classification of speech and nonlinguistic vocalisations. The model consists of multiple identical streams, one for each modality, which extract features directly from mouth regions and spectrograms. The temporal dynamics in each stream/modality are modeled by a BLSTM and the fusion of multiple streams/modalities takes place via another BLSTM. An absolute improvement of 1.9% in the mean F1 of 4 nonlingusitic vocalisations over audio-only classification is reported on the AVIC database. At the same time, the proposed end-to-end audiovisual fusion system improves the state-of-the-art performance on the AVIC database leading to a 9.7% absolute increase in the mean F1 measure. We also perform audiovisual speech recognition experiments on the OuluVS2 database using different views of the mouth, frontal to profile. The proposed audiovisual system significantly outperforms the audio-only model for all views when the acoustic noise is high.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑