arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.22374cs.CL
收藏

选择性对比学习用于无 gloss 手语翻译

Selective Contrastive Learning For Gloss Free Sign Language Translation

School of Informatics, Xiamen University, China(厦门大学信息学院);Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China(福建省 Fujian-Taiwan 非物质文化遗产数字保护与智能处理重点实验室,文化部,中国);National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China(教育与教学媒体语言资源监测与研究中心,厦门大学,中国)

Changhao Lai, Rui Zhao, Xuewen Zhong, Jinsong Su, Yidong Chen

AI总结:

本文提出选择性对比学习方法,通过动态选择负样本提升手语翻译的跨模态对齐效果,减少噪声干扰。

摘要:

手语翻译(SLT)将连续手语视频转换为口语文本,但因视觉符号与书面文本的固有模态不匹配,尤其在无 gloss 设置下仍具挑战性。最近的 SLT 系统越来越多地采用类似 CLIP 的视觉-语言预训练(VLP)进行跨模态对齐,但随机批次对比提供少量依赖批次的负样本,可能将语义相似(或甚至相同)的对误标为负样本,引入噪声且可能不一致的对齐监督。本文首先进行初步轨迹分析,追踪负视频-文本相似度随训练的变化。结果表明,只有少数负样本表现出期望的行为,即被一致推远,而其余负样本则表现出异质性和非递增的相似度动态,表明随机批次负样本对有效对齐往往无用。受此启发,我们提出用于 SLT 的选择性对比学习(SCL-SLT)方法,采用配对选择(PS)策略。PS 使用参考检查点的相似度动态评分候选负样本,并通过课程学习构建 mini-batches,逐步强调更困难的负样本,从而加强对比监督并减少噪声或语义无效负样本的影响。

英文摘要:

Sign language translation (SLT) converts continuous sign videos into spoken-language text, yet it remains challenging due to the intrinsic modality mismatch between visual signs and written text, particularly in gloss-free settings. Recent SLT systems increasingly adopt CLIP-like Vision-Language pretraining (VLP) for cross-modal alignment, but the random in-batch contrast provides few, batch-dependent negatives and may mislabel semantically similar (or even identical) pairs as negatives, introducing noisy and potentially inconsistent alignment supervision. In this work, we first conduct a preliminary trajectory-based analysis that tracks negative video-text similarity over training. The results show that only a small subset of negatives exhibits the desired behavior of being consistently pushed away, while the remaining negatives display heterogeneous and often non-decreasing similarity dynamics, suggesting that random in-batch negatives are frequently uninformative for effective alignment. Inspired by this, we propose Selective Contrastive Learning for SLT (SCL-SLT) with a Pair Selection (PS) strategy. PS scores candidate negatives using similarity dynamics from reference checkpoints and constructs mini-batches via a curriculum that progressively emphasizes more challenging negatives, thereby strengthening contrastive supervision while reducing the influence of noisy or semantically invalid negatives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑