arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.19488cs.CV
收藏

MM-WLAuslan:多视角多模态词级澳大利亚手语识别数据集

MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset

The University of Queensland(昆士兰大学)

Xin Shen, Heming Du, Hongwei Sheng, Shuyun Wang, Hui Chen, Huiqiang Chen, Zhuojie Wu, Xiaobiao Du, Jiaying Ying, Ruihan Lu, Qingzheng Xu, Xin Yu

AI总结:

本文构建首个大规模多视角多模态词级澳大利亚手语数据集MM-WLAuslan,并通过多视角、跨相机和跨视角基准验证其挑战性。

摘要:

孤立手语识别(ISLR)专注于识别单个手语词汇。考虑到不同地理区域手语的多样性,开发针对特定地区的ISLR数据集对于支持交流与研究至关重要。Auslan作为澳大利亚特有的手语,在ISLR任务上仍缺乏专门的大规模词级数据集。为填补这一空白,我们构建了首个大规模多视角多模态词级澳大利亚手语识别数据集,命名为MM-WLAuslan。与其他公开可用数据集相比,MM-WLAuslan展现出三个显著优势:(1)数据量最大,(2)词汇表最广泛,(3)多模态相机视角最多样。具体而言,我们在摄影棚环境中录制了超过28.2万条手语视频,涵盖由73名手语者呈现的3,215个常用Auslan词汇。此外,我们的拍摄系统包含两种不同类型的相机,即三台Kinect-V2相机和一台RealSense相机。我们将相机以半球形布置在模特前半周周围,并使用全部四台相机同时录制视频。此外,我们在MM-WLAuslan上使用面向多种多模态ISLR设置的最先进方法进行基准测试,包括多视角、跨相机和跨视角设置。实验结果表明,MM-WLAuslan是一个具有挑战性的ISLR数据集,我们希望该数据集能够促进Auslan的发展以及全球手语的进步。所有数据集和基准均可在MM-WLAuslan获取。

英文摘要:

Isolated Sign Language Recognition (ISLR) focuses on identifying individual sign language glosses. Considering the diversity of sign languages across geographical regions, developing region-specific ISLR datasets is crucial for supporting communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale word-level dataset for the ISLR task. To fill this gap, we curate \underline{\textbf{the first}} large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan. Compared to other publicly available datasets, MM-WLAuslan exhibits three significant advantages: (1) the largest amount of data, (2) the most extensive vocabulary, and (3) the most diverse of multi-modal camera views. Specifically, we record 282K+ sign videos covering 3,215 commonly used Auslan glosses presented by 73 signers in a studio environment. Moreover, our filming system includes two different types of cameras, i.e., three Kinect-V2 cameras and a RealSense camera. We position cameras hemispherically around the front half of the model and simultaneously record videos using all four cameras. Furthermore, we benchmark results with state-of-the-art methods for various multi-modal ISLR settings on MM-WLAuslan, including multi-view, cross-camera, and cross-view. Experiment results indicate that MM-WLAuslan is a challenging ISLR dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide. All datasets and benchmarks are available at MM-WLAuslan.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑