SignCLIP:通过对比学习连接文本与手语
SignCLIP: Connecting Text and Sign Language by Contrastive Learning
AI总结:
本研究提出SignCLIP方法,复用CLIP将口语文本与手语视频映射到同一空间,在大规模多语言手语数据集上预训练后,在跨模态检索及少样本下游手语任务中表现优异,还为语言学研究提供了新视角。
摘要:
我们提出SignCLIP,该方法重新利用CLIP(对比语言-图像预训练),将口语化文本和手语视频这两类模态截然不同的自然语言映射到同一空间。SignCLIP是一种高效方法,可从大规模多语言视频-文本对中学习手语处理所需的有用视觉表示,无需针对规模通常有限的特定任务或手语直接优化。\n 我们在Spreadthesign(一个知名手语词典,包含约50万条视频片段,覆盖多达44种手语)上预训练SignCLIP,并在多个下游数据集上对其进行评估。SignCLIP在域内手语识别中表现出色,文本到视频/视频到文本的检索准确率可观。在经过必要的少样本提示或微调后,它在孤立手语识别等域外下游任务上也具备竞争力。\n 我们分析了由口语化文本和手语姿态构成的潜在空间,这提供了额外的语言学洞见。我们的代码和模型已公开可用。
英文摘要:
We present SignCLIP, which re-purposes CLIP (Contrastive Language-Image Pretraining) to project spoken language text and sign language videos, two classes of natural languages of distinct modalities, into the same space. SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs, without directly optimizing for a specific task or sign language which is often of limited size. We pretrain SignCLIP on Spreadthesign, a prominent sign language dictionary consisting of ~500 thousand video clips in up to 44 sign languages, and evaluate it with various downstream datasets. SignCLIP discerns in-domain signing with notable text-to-video/video-to-text retrieval accuracy. It also performs competitively for out-of-domain downstream tasks such as isolated sign language recognition upon essential few-shot prompting or fine-tuning. We analyze the latent space formed by the spoken language text and sign language poses, which provides additional linguistic insights. Our code and models are openly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。