arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.10428cs.CV
收藏

基于全局-局部增强网络的NMF感知手语识别

Global-local Enhancement Network for NMFs-aware Sign Language Recognition

University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence(人工智能研究院)

Hezhen Hu, Wengang Zhou, Junfu Pu, Houqiang Li

摘要:

手语识别(SLR)是一个具有挑战性的问题,涉及复杂的手动特征,即,手势和细粒度非手动特征(NMF),即,虽然手势特征占主导地位,但非手势特征也在手势词的表达中起着重要作用。具体来说,许多手势语由于非手动特征而传达不同的含义,即使它们共享相同的手势。这种模糊性给手语识别带来了很大的挑战。为了解决上述问题,我们提出了一个简单而有效的架构称为全球本地增强网络(GLE-Net),包括两个相互促进的流对不同的关键方面的SLR。在这两个流中,一个流捕获全局上下文关系,而另一个流捕获区分性细粒度线索。此外,由于缺乏明确关注此类特征的数据集,我们引入了第一个非手动特征感知的孤立中国手语数据集~(NMFs-CSL),其总词汇量为1,067个日常生活中的手语词。在NMFs-CSL和SLR 500数据集上的实验证明了该方法的有效性。

英文摘要:

Sign language recognition (SLR) is a challenging problem, involving complex manual features, i.e., hand gestures, and fine-grained non-manual features (NMFs), i.e., facial expression, mouth shapes, etc. Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-local Enhancement Network (GLE-Net), including two mutually promoted streams towards different crucial aspects of SLR. Of the two streams, one captures the global contextual relationship, while the other stream captures the discriminative fine-grained cues. Moreover, due to the lack of datasets explicitly focusing on this kind of features, we introduce the first non-manual-features-aware isolated Chinese sign language dataset~(NMFs-CSL) with a total vocabulary size of 1,067 sign words in daily life. Extensive experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑