带有3D-2D- CNN BLSTM-HMM和单词CTC模型的LipReading
LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC models
摘要:
近年来,基于深度学习的机器唇读已经变得突出。为此,已经提出了几种架构,如LipNet,LCANet等,与传统的唇读DNN-HMM混合系统相比,它们在DCT特征上训练的性能非常好。在这项工作中,我们提出了一种具有瓶颈层的3D-2D-CNN-BLSTM网络的更简单的架构。我们还提出了两种不同的唇读方法在这个架构的分析。在第一种方法中,3D-2D-CNN-BLSTM网络使用CTC字符损失(ch-CTC)进行训练。然后,在传统的ASR训练管道中,在瓶颈唇特征(从3D-2D-CNN-BLSTM ch-CTC网络中提取)上训练BLSTM-HMM模型。在第二种方法中,相同的3D-2D-CNN-BLSTM网络使用单词标签上的CTC损失(w-CTC)进行训练。第一种方法表明,瓶颈特征比DCT特征表现得更好。使用第二种方法在网格语料库的见人测试集上,我们报告了$1.3\%$ WER -一个相对于LCANet的$55\%$改进。在看不见的说话人测试集上,我们报告了$8.6\%$ WER,这是相对于LipNet的$24.5\%$改进。我们还验证了第二个数据集的$81$扬声器,我们收集的方法。最后,我们还讨论了特征重复对BLSTM-HMM模型性能的影响。
英文摘要:
In recent years, deep learning based machine lipreading has gained prominence. To this end, several architectures such as LipNet, LCANet and others have been proposed which perform extremely well compared to traditional lipreading DNN-HMM hybrid systems trained on DCT features. In this work, we propose a simpler architecture of 3D-2D-CNN-BLSTM network with a bottleneck layer. We also present analysis of two different approaches for lipreading on this architecture. In the first approach, 3D-2D-CNN-BLSTM network is trained with CTC loss on characters (ch-CTC). Then BLSTM-HMM model is trained on bottleneck lip features (extracted from 3D-2D-CNN-BLSTM ch-CTC network) in a traditional ASR training pipeline. In the second approach, same 3D-2D-CNN-BLSTM network is trained with CTC loss on word labels (w-CTC). The first approach shows that bottleneck features perform better compared to DCT features. Using the second approach on Grid corpus' seen speaker test set, we report $1.3\%$ WER - a $55\%$ improvement relative to LCANet. On unseen speaker test set we report $8.6\%$ WER which is $24.5\%$ improvement relative to LipNet. We also verify the method on a second dataset of $81$ speakers which we collected. Finally, we also discuss the effect of feature duplication on BLSTM-HMM model performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。