arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.14768cs.CV
收藏

无注释手语翻译:从视觉-语言预训练中改进

Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining

MUST(澳门科技大学);UCAS(中国科学院大学);MAIS, CASIA(中国科学院自动化研究所模式识别国家重点实验室(注:MAIS为CASIA下属部门,按要求提取至研究院级,此处取CASIA对应中文名:中国科学院自动化研究所));Universitat de Barcelona(巴塞罗那大学);Computer Vision Center(计算机视觉中心);AAU(奥尔堡大学);CAIR, HKISI, CAS(中国科学院香港创新研究院人工智能与机器人创新研究所(注:按要求提取至研究院级,取CAS对应中文名:中国科学院;HKISI为中国科学院香港创新研究院;CAIR为其下属机构,此处提取至研究院级:中国科学院香港创新研究院))

Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang

AI总结:

提出基于视觉-语言预训练的无gloss手语翻译方法GFSLT-VLP,通过结合CLIP与掩码自监督学习及端到端编码器-解码器架构,在无需gloss注释的情况下显著提升了翻译性能。

摘要:

手语翻译(SLT)因其跨领域的性质而是一项具有挑战性的任务,涉及将视觉-手势语言翻译为文本。许多先前的方法采用中间表示,即gloss序列,来促进SLT,从而将其转化为手语识别(SLR)随后进行手语翻译(SLT)的两阶段任务。然而,gloss注释手语数据的稀缺,加上中级gloss表示中的信息瓶颈,阻碍了SLT任务的进一步发展。为了解决这一挑战,我们提出了一种基于视觉-语言预训练(GFSLT-VLP)的新型无Gloss SLT方法,该方法通过从预训练模型中继承面向语言的先验知识来改进SLT,无需任何gloss注释辅助。我们的方法包括两个阶段:(i)将对比语言-图像预训练(CLIP)与掩码自监督学习相结合,创建预任务,以弥合视觉和文本表示之间的语义差距并恢复掩码句子;(ii)构建一个具有类似编码器-解码器结构的端到端架构,该架构从第一阶段继承预训练的视觉编码器和文本解码器的参数。这些新颖设计的无缝结合形成了稳健的手语表示,并显著改进了无gloss手语翻译。特别是,与最先进的无gloss SLT方法相比,我们在PHOENIX14T数据集(>+5)和CSL-Daily数据集(>+3)上的BLEU-4分数取得了前所未有的改进。此外,与大多数基于gloss的方法相比,我们的方法在PHOENIX14T数据集上也取得了有竞争力的结果。我们的代码可在https://github.com/zhoubenjia/GFSLT-VLP获取。

英文摘要:

Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a two-stage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods. Our code is available at https://github.com/zhoubenjia/GFSLT-VLP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑