arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.03467cs.CLcs.AI
收藏

Continuous Saudi Sign Language Recognition: A Vision Transformer Approach(连续沙特手语识别:一种视觉Transformer方法)

Continuous Saudi Sign Language Recognition: A Vision Transformer Approach

Department of Computer Science, Faculty of Computing and Information Technology(计算机科学系,计算与信息科技学院)

Soukeina Elhassen, Lama Al Khuzayem, Areej Alhothali, Ohoud Alzamzami, Nahed Alowaidi

AI总结:

针对沙特手语资源匮乏、现有研究多聚焦孤立词的问题,本研究构建首个连续沙特手语数据集KAU-CSSL,并提出结合预训练ResNet-18与双向LSTM Transformer编码器的识别模型,在两种模式下均取得良好效果。

摘要:

手语(SL)是听障和聋人群体的重要沟通形式,帮助他们融入更广泛的社会。尽管手语意义重大,但公众对手语的认知有限,往往导致听障群体难以公平获得教育和职业机会,进而加剧社会排斥,这一问题在沙特阿拉伯尤为突出——当地有超过8.4万人将沙特手语(SSL)作为主要沟通方式。虽然部分技术手段已在改善听障人士沟通状况方面发挥了作用,但目前仍迫切需要更精准、更可靠的翻译技术,尤其是针对沙特手语这类阿拉伯语手语变体的技术。当前大多数前沿解决方案主要聚焦于非阿拉伯语手语,导致阿拉伯语手语(特别是沙特手语)相关资源严重匮乏。阿拉伯语本身的复杂性,以及现有手语数据集多为孤立词数据集(聚焦单个词汇而非连续语句)的普遍情况,是造成这一缺口的重要原因。为填补这一空白,本研究是沙特手语资源建设进程中的重要一步。为此,我们推出了首个连续沙特手语数据集KAU-CSSL,该数据集以完整语句为核心,旨在为后续研究提供支持,并助力开发更先进的沙特手语识别与翻译系统。此外,我们提出了一种基于Transformer的模型:采用预训练ResNet-18提取空间特征,搭配结合双向LSTM的Transformer编码器处理时序依赖关系,该模型在依赖手语者模式下准确率达99.02%,在独立手语者模式下准确率达77.71%。这项研究不仅为优化沙特手语群体的沟通工具开辟了道路,也为更广泛的手语领域研究做出了重要贡献。

英文摘要:

Sign language (SL) is an essential communication form for hearing-impaired and deaf people, enabling engagement within the broader society. Despite its significance, limited public awareness of SL often leads to inequitable access to educational and professional opportunities, thereby contributing to social exclusion, particularly in Saudi Arabia, where over 84,000 individuals depend on Saudi Sign Language (SSL) as their primary form of communication. Although certain technological approaches have helped to improve communication for individuals with hearing impairments, there continues to be an urgent requirement for more precise and dependable translation techniques, especially for Arabic sign language variants like SSL. Most state-of-the-art solutions have primarily focused on non-Arabic sign languages, resulting in a considerable absence of resources dedicated to Arabic sign language, specifically SSL. The complexity of the Arabic language and the prevalence of isolated sign language datasets that concentrate on individual words instead of continuous speech contribute to this issue. To address this gap, our research represents an important step in developing SSL resources. To address this, we introduce the first continuous Saudi Sign Language dataset called KAU-CSSL, focusing on complete sentences to facilitate further research and enable sophisticated recognition systems for SSL recognition and translation. Additionally, we propose a transformer-based model, utilizing a pretrained ResNet-18 for spatial feature extraction and a Transformer Encoder with Bidirectional LSTM for temporal dependencies, achieving 99.02\% accuracy at signer dependent mode and 77.71\% accuracy at signer independent mode. This development leads the way to not only improving communication tools for the SSL community but also making a substantial contribution to the wider field of sign language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑