MM-WLAuslan:多视角多模态词级澳大利亚手语识别数据集
MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset
AI总结:
本文构建首个大规模多视角多模态词级澳大利亚手语数据集MM-WLAuslan,并通过多视角、跨相机和跨视角基准验证其挑战性。
摘要:
孤立手语识别(ISLR)专注于识别单个手语词汇。考虑到不同地理区域手语的多样性,开发针对特定地区的ISLR数据集对于支持交流与研究至关重要。Auslan作为澳大利亚特有的手语,在ISLR任务上仍缺乏专门的大规模词级数据集。为填补这一空白,我们构建了首个大规模多视角多模态词级澳大利亚手语识别数据集,命名为MM-WLAuslan。与其他公开可用数据集相比,MM-WLAuslan展现出三个显著优势:(1)数据量最大,(2)词汇表最广泛,(3)多模态相机视角最多样。具体而言,我们在摄影棚环境中录制了超过28.2万条手语视频,涵盖由73名手语者呈现的3,215个常用Auslan词汇。此外,我们的拍摄系统包含两种不同类型的相机,即三台Kinect-V2相机和一台RealSense相机。我们将相机以半球形布置在模特前半周周围,并使用全部四台相机同时录制视频。此外,我们在MM-WLAuslan上使用面向多种多模态ISLR设置的最先进方法进行基准测试,包括多视角、跨相机和跨视角设置。实验结果表明,MM-WLAuslan是一个具有挑战性的ISLR数据集,我们希望该数据集能够促进Auslan的发展以及全球手语的进步。所有数据集和基准均可在MM-WLAuslan获取。
英文摘要:
Isolated Sign Language Recognition (ISLR) focuses on identifying individual sign language glosses. Considering the diversity of sign languages across geographical regions, developing region-specific ISLR datasets is crucial for supporting communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale word-level dataset for the ISLR task. To fill this gap, we curate \underline{\textbf{the first}} large-scale Multi-view Multi-modal Word-Level Australian Sign Language recognition dataset, dubbed MM-WLAuslan. Compared to other publicly available datasets, MM-WLAuslan exhibits three significant advantages: (1) the largest amount of data, (2) the most extensive vocabulary, and (3) the most diverse of multi-modal camera views. Specifically, we record 282K+ sign videos covering 3,215 commonly used Auslan glosses presented by 73 signers in a studio environment. Moreover, our filming system includes two different types of cameras, i.e., three Kinect-V2 cameras and a RealSense camera. We position cameras hemispherically around the front half of the model and simultaneously record videos using all four cameras. Furthermore, we benchmark results with state-of-the-art methods for various multi-modal ISLR settings on MM-WLAuslan, including multi-view, cross-camera, and cross-view. Experiment results indicate that MM-WLAuslan is a challenging ISLR dataset, and we hope this dataset will contribute to the development of Auslan and the advancement of sign languages worldwide. All datasets and benchmarks are available at MM-WLAuslan.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。