arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16394cs.CV
收藏

SEDS:用于手语检索的语义增强双流编码器

SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval

University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心);Merchants Union Consumer Finance Company Limited(招商局消费金融有限公司)

Longtao Jiang, Min Wang, Zecheng Li, Yao Fang, Wengang Zhou, Houqiang Li

AI总结:

提出手语表示框架SEDS,融合Pose和RGB双模态表征局部与全局信息,通过跨词汇注意力融合模块和细粒度匹配目标实现端到端训练,在多个数据集上显著优于现有方法。

摘要:

与传统视频检索不同,手语检索更偏向于理解视频片段中包含的人类动作的语义信息。以往的工作通常仅对RGB视频进行编码以获取高级语义特征,导致局部动作细节淹没在大量视觉信息冗余中。此外,现有的基于RGB的手语检索工作在端到端训练中面临密集视觉数据嵌入带来的巨大内存开销,因此转而采用离线RGB编码器,导致特征表示次优。为了解决这些问题,我们提出了一种名为语义增强双流编码器(SEDS)的新型手语表示框架,该框架融合了Pose和RGB模态,以表示手语视频的局部和全局信息。具体而言,Pose编码器嵌入与人体关节对应的关键点坐标,有效捕捉详细的动作特征。为了更好地实现两种视频模态的上下文感知融合,我们提出了跨词汇注意力融合(CGAF)模块,以聚合来自模态内和模态间具有相似语义信息的相邻片段特征。此外,开发了Pose-RGB细粒度匹配目标,通过细粒度双流特征的上下文匹配来增强聚合的融合特征。除离线RGB编码器外,整个框架仅包含可学习的轻量级网络,可进行端到端训练。大量实验表明,我们的框架在多个数据集上显著优于现有最先进的方法。

英文摘要:

Different from traditional video retrieval, sign language retrieval is more biased towards understanding the semantic information of human actions contained in video clips. Previous works typically only encode RGB videos to obtain high-level semantic features, resulting in local action details drowned in a large amount of visual information redundancy. Furthermore, existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training, and adopt offline RGB encoder instead, leading to suboptimal feature representation. To address these issues, we propose a novel sign language representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which integrates Pose and RGB modalities to represent the local and global information of sign language videos. Specifically, the Pose encoder embeds the coordinates of keypoints corresponding to human joints, effectively capturing detailed action features. For better context-aware fusion of two video modalities, we propose a Cross Gloss Attention Fusion (CGAF) module to aggregate the adjacent clip features with similar semantic information from intra-modality and inter-modality. Moreover, a Pose-RGB Fine-grained Matching Objective is developed to enhance the aggregated fusion feature by contextual matching of fine-grained dual-stream features. Besides the offline RGB encoder, the whole framework only contains learnable lightweight networks, which can be trained end-to-end. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods on various datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑