arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.13023cs.CV
收藏

利用一致性约束与签名者移除改进连续手语识别

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

The Hong Kong University of Science and Technology(香港科技大学)

Ronglai Zuo, Brian Mak

AI总结:

本文针对连续手语识别训练不足问题,提出空间注意力一致性、句子嵌入一致性约束及签名者移除三个辅助任务,在五个基准上取得领先性能。

摘要:

大多数基于深度学习的连续手语识别(CSLR)模型共享相似的骨干结构,包括视觉模块、序列模块和对齐模块。然而,由于训练样本有限,连接主义时序分类损失可能无法充分训练此类CSLR骨干。在本文中,我们提出三个辅助任务来增强CSLR骨干。第一个任务从一致性的角度增强对训练不足问题敏感的视觉模块。具体而言,由于手语信息主要包含在签名者的面部表情和手部运动中,我们开发了一个关键点引导的空间注意力模块,以强制视觉模块关注信息丰富的区域,即空间注意力一致性。第二,注意到视觉模块和序列模块的输出特征都表示同一个句子,为了更好地利用骨干的能力,在视觉模块和序列模块之间施加句子嵌入一致性约束,以增强两种特征的表示能力。我们将使用上述辅助任务训练的CSLR模型命名为一致性增强CSLR,该模型在签名者相关数据集上表现良好,在这些数据集中所有签名者在训练和测试期间都会出现。为了使其对签名者无关设置更加鲁棒,进一步提出了基于特征解耦的签名者移除模块,以从骨干中移除签名者信息。进行了广泛的消融研究以验证这些辅助任务的有效性。更值得注意的是,借助基于Transformer的骨干,我们的模型在五个基准数据集PHOENIX-2014、PHOENIX-2014-T、PHOENIX-2014-SI、CSL和CSL-Daily上取得了最先进或具有竞争力的性能。代码和模型可在https://github.com/2000ZRL/LCSA_C2SLR_SRM获取。

英文摘要:

Most deep-learning-based continuous sign language recognition (CSLR) models share a similar backbone consisting of a visual module, a sequential module, and an alignment module. However, due to limited training samples, a connectionist temporal classification loss may not train such CSLR backbones sufficiently. In this work, we propose three auxiliary tasks to enhance the CSLR backbones. The first task enhances the visual module, which is sensitive to the insufficient training problem, from the perspective of consistency. Specifically, since the information of sign languages is mainly included in signers' facial expressions and hand movements, a keypoint-guided spatial attention module is developed to enforce the visual module to focus on informative regions, i.e., spatial attention consistency. Second, noticing that both the output features of the visual and sequential modules represent the same sentence, to better exploit the backbone's power, a sentence embedding consistency constraint is imposed between the visual and sequential modules to enhance the representation power of both features. We name the CSLR model trained with the above auxiliary tasks as consistency-enhanced CSLR, which performs well on signer-dependent datasets in which all signers appear during both training and testing. To make it more robust for the signer-independent setting, a signer removal module based on feature disentanglement is further proposed to remove signer information from the backbone. Extensive ablation studies are conducted to validate the effectiveness of these auxiliary tasks. More remarkably, with a transformer-based backbone, our model achieves state-of-the-art or competitive performance on five benchmarks, PHOENIX-2014, PHOENIX-2014-T, PHOENIX-2014-SI, CSL, and CSL-Daily. Code and Models are available at https://github.com/2000ZRL/LCSA_C2SLR_SRM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑