arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.04152cs.CV
收藏

通过手语词典扩大符号识别

Scaling up sign spotting through sign language dictionaries

University of Oxford(牛津大学);École des Ponts(巴黎桥路学院);Univ Gustave Eiffel(古斯塔夫·埃菲尔大学);CNRS(法国国家科学研究中心);University of Cambridge(剑桥大学)

Gül Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, Andrew Zisserman

摘要:

这项工作的重点是$\textit{sign spotting}$ -给定一个视频的一个孤立的标志,我们的任务是确定$\textit{whether}$和$\textit{where}$它已签署了一个连续的,共同阐述的手语视频。为了实现这个标志识别任务,我们通过以下方式使用多种类型的可用监督来训练模型:(1)$\textit{watching}$现有的片段,其使用口型提示稀疏地标记;(2)$\textit{reading}$关联的字幕(签名内容的现成翻译),提供额外的$\textit{weak-supervision}$ ;(3)视觉手语词典中的$\textit{looking up}$词(没有共同发音的标记示例),以实现新的符号识别。这三个任务被集成到一个统一的学习框架,使用噪声对比估计和多实例学习的原则。我们验证了我们的方法的有效性低杆标志定位基准。此外,我们贡献了一个机器可读的英国手语(BSL)字典数据集的孤立的迹象,BSLDict,以促进这项任务的研究。数据集,模型和代码可在我们的项目页面上找到。

英文摘要:

The focus of this work is $\textit{sign spotting}$ - given a video of an isolated sign, our task is to identify $\textit{whether}$ and $\textit{where}$ it has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1) $\textit{watching}$ existing footage which is sparsely labelled using mouthing cues; (2) $\textit{reading}$ associated subtitles (readily available translations of the signed content) which provide additional $\textit{weak-supervision}$; (3) $\textit{looking up}$ words (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs, BSLDict, to facilitate study of this task. The dataset, models and code are available at our project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑