SignMouth：通过多模态对比融合利用口型线索进行手语翻译
SignMouth: Leveraging Mouthing Cues for Sign Language Translation by Multimodal Contrastive Fusion
AI总结：
针对现有手语翻译多忽略口型非手动线索的问题，本文提出SignClip框架，融合手势与口型特征并引入分层对比学习实现多模态对齐，在两个基准数据集上取得了更优的翻译效果。
摘要：
手语翻译（SLT）旨在从手语视频中翻译出自然语言，是实现包容性沟通的重要桥梁。尽管近期研究借助强大的视觉骨干网络和大语言模型取得了进展，但大多数方法主要关注手动信号（手势），往往忽略口型等非手动线索。事实上，口型在手语中传递着关键的语言信息，对区分视觉上相似的手语符号至关重要。本文提出SignClip——一种提升手语翻译准确率的新型框架，它融合手动与非手动线索，具体为空间手势特征与唇部运动特征。此外，SignClip引入了带有多层对齐目标的分层对比学习框架，确保手语-唇部以及视觉-文本模态间的语义一致性。在PHOENIX14T和How2Sign两个基准数据集上的大量实验证明了该方法的优越性。例如，在PHOENIX14T数据集的无 Gloss 设置下，SignClip超越了此前的最优模型SpaMo，将BLEU-4从24.32提升至24.71，ROUGE从46.57提升至48.38。
英文摘要：
Sign language translation (SLT) aims to translate natural language from sign language videos, serving as a vital bridge for inclusive communication. While recent advances leverage powerful visual backbones and large language models, most approaches mainly focus on manual signals (hand gestures) and tend to overlook non-manual cues like mouthing. In fact, mouthing conveys essential linguistic information in sign languages and plays a crucial role in disambiguating visually similar signs. In this paper, we propose SignClip, a novel framework to improve the accuracy of sign language translation. It fuses manual and non-manual cues, specifically spatial gesture and lip movement features. Besides, SignClip introduces a hierarchical contrastive learning framework with multi-level alignment objectives, ensuring semantic consistency across sign-lip and visual-text modalities. Extensive experiments on two benchmark datasets, PHOENIX14T and How2Sign, demonstrate the superiority of our approach. For example, on PHOENIX14T, in the Gloss-free setting, SignClip surpasses the previous state-of-the-art model SpaMo, improving BLEU-4 from 24.32 to 24.71, and ROUGE from 46.57 to 48.38.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
