BSL-1K:使用口形提示扩展共同表达的手语识别
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
摘要:
最近在细粒度手势和动作分类以及机器翻译方面的进展,指出了自动手语识别成为现实的可能性。在实现这一目标方面取得进展的一个关键障碍是缺乏适当的训练数据,这是由于标志注释的高度复杂性和合格注释者的供应有限。在这项工作中,我们介绍了一种新的可扩展的方法来数据汇集的标志识别连续视频。我们利用弱对齐字幕和关键字识别方法,在1,000小时的视频中自动定位1,000个标志的词汇。我们做出了以下贡献:(1)我们展示了如何使用来自签名者的口形线索来从视频数据中获得高质量的注释-结果是BSL-1 K数据集,一个前所未有的英国手语(BSL)标志的集合;(2)我们证明了我们可以用BSL-1 K来训练强标志识别模型,这些模型还为其他手语和基准提供了出色的预训练-我们在MSASL和WLASL基准上都超过了最先进的水平。最后,(3)我们提出了新的大规模评估集的任务的标志识别和标志发现,并提供基线,我们希望这将有助于刺激在这一领域的研究。
英文摘要:
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A key stumbling block in making progress towards this goal is a lack of appropriate training data, stemming from the high complexity of sign annotation and a limited supply of qualified annotators. In this work, we introduce a new scalable approach to data collection for sign recognition in continuous videos. We make use of weakly-aligned subtitles for broadcast footage together with a keyword spotting method to automatically localise sign-instances for a vocabulary of 1,000 signs in 1,000 hours of video. We make the following contributions: (1) We show how to use mouthing cues from signers to obtain high-quality annotations from video data - the result is the BSL-1K dataset, a collection of British Sign Language (BSL) signs of unprecedented scale; (2) We show that we can use BSL-1K to train strong sign recognition models for co-articulated signs in BSL and that these models additionally form excellent pretraining for other sign languages and benchmarks - we exceed the state of the art on both the MSASL and WLASL benchmarks. Finally, (3) we propose new large-scale evaluation sets for the tasks of sign recognition and sign spotting and provide baselines which we hope will serve to stimulate research in this area.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。