arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.09949cs.CVcs.CL
收藏

C${^2}$RL:面向无 gloss 手语翻译与检索的内容与上下文表示学习

C${^2}$RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval

State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Macau University of Science and Technology(澳门科技大学);NIO(蔚来汽车)

Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, Du Zhang

AI总结:

针对无gloss手语表示学习中非单调对齐难以捕捉精细特征的问题,提出融合隐式内容学习与显式上下文学习的C²RL预训练范式,在多个手语翻译与检索数据集上实现显著性能提升并刷新基准。

摘要:

手语表示学习(SLRL)对手语翻译(SLT)、手语检索(SLRet)等一系列手语相关下游任务至关重要。近年来,众多基于 gloss 和无 gloss 的 SLRL 方法被相继提出,展现出可观的性能。其中,无 gloss 方法无需依赖 gloss 标注,具备强可扩展性,应用前景广阔。但目前该方法仍未达到最优效果,核心难点在于难以编码手语视频复杂、上下文敏感的特征,尤其在采用非单调视频-文本对齐策略时,难以精准识别关键手语特征。\n为此,本文提出一种创新的无 gloss SLRL 预训练范式 C${^2}$RL。具体而言,该方法并非仅通过视频与文本的非单调语义对齐来学习面向语言的手语特征,而是重点强化 SLRL 的两个核心维度:隐式内容学习(ICL)与显式上下文学习(ECL)。ICL 聚焦手语的交流内容层面,捕捉手语的细微差异、强调方式、时序节奏等细节;ECL 则侧重于理解手语的上下文语义,并将其转换为对应的通顺句子。\n尽管方法框架简洁,大量实验证实,ICL 与 ECL 的联合优化可学习到鲁棒的手语表示,在无 gloss SLT 和 SLRet 任务上均实现显著性能提升。值得注意的是,C${^2}$RL 将 BLEU-4 分数在 P14T 数据集上提升 +5.3,在 CSL-daily 上提升 +10.6,在 OpenASL 上提升 +6.2,在 How2Sign 上提升 +1.3;同时将 R@1 分数在 P14T 上提升 +8.3,在 CSL-daily 上提升 +14.4,在 How2Sign 上提升 +5.9。此外,本文还为 OpenASL 数据集的 SLRet 任务设立了新的基准。

英文摘要:

Sign Language Representation Learning (SLRL) is crucial for a range of sign language-related downstream tasks such as Sign Language Translation (SLT) and Sign Language Retrieval (SLRet). Recently, many gloss-based and gloss-free SLRL methods have been proposed, showing promising performance. Among them, the gloss-free approach shows promise for strong scalability without relying on gloss annotations. However, it currently faces suboptimal solutions due to challenges in encoding the intricate, context-sensitive characteristics of sign language videos, mainly struggling to discern essential sign features using a non-monotonic video-text alignment strategy. Therefore, we introduce an innovative pretraining paradigm for gloss-free SLRL, called C${^2}$RL, in this paper. Specifically, rather than merely incorporating a non-monotonic semantic alignment of video and text to learn language-oriented sign features, we emphasize two pivotal aspects of SLRL: Implicit Content Learning (ICL) and Explicit Context Learning (ECL). ICL delves into the content of communication, capturing the nuances, emphasis, timing, and rhythm of the signs. In contrast, ECL focuses on understanding the contextual meaning of signs and converting them into equivalent sentences. Despite its simplicity, extensive experiments confirm that the joint optimization of ICL and ECL results in robust sign language representation and significant performance gains in gloss-free SLT and SLRet tasks. Notably, C${^2}$RL improves the BLEU-4 score by +5.3 on P14T, +10.6 on CSL-daily, +6.2 on OpenASL, and +1.3 on How2Sign. It also boosts the R@1 score by +8.3 on P14T, +14.4 on CSL-daily, and +5.9 on How2Sign. Additionally, we set a new baseline for the OpenASL dataset in the SLRet task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑