用于唇读的具有记忆增强横向Transformers的SpotFast网络
SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading
摘要：
本文提出了一种新的用于单词级唇读的深度学习架构。以前的工作表明，有可能将预训练的深度3D卷积神经网络作为前端特征提取器。我们介绍了一个SpotFast网络，一个变种的国家的最先进的动作识别的SlowFast网络，它利用时间窗口作为一个点的途径和所有帧作为一个快速的途径。我们进一步将内存增强横向Transformers学习分类的顺序功能。我们在LRW数据集上评估了所提出的模型。实验表明，我们提出的模型优于各种最先进的模型，并且结合内存增强的横向Transformers使SpotFast网络提高了3.7%。
英文摘要：
This paper presents a novel deep learning architecture for word-level lipreading. Previous works suggest a potential for incorporating a pretrained deep 3D Convolutional Neural Networks as a front-end feature extractor. We introduce a SpotFast networks, a variant of the state-of-the-art SlowFast networks for action recognition, which utilizes a temporal window as a spot pathway and all frames as a fast pathway. We further incorporate memory augmented lateral transformers to learn sequential features for classification. We evaluate the proposed model on the LRW dataset. The experiments show that our proposed model outperforms various state-of-the-art models and incorporating the memory augmented lateral transformers makes a 3.7% improvement to the SpotFast networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
