arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.04367cs.CV
收藏

微调视频变换器用于词级孟加拉手语:分类任务的比较分析

Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Tasks

Systems and Software Lab (SSL), Department of CSE, Islamic University of Technology (IUT)(计算机科学与软件系,伊斯兰科技大学(IUT)系统与软件实验室)

Jubayer Ahmed Bhuiyan Shawon, Hasan Mahmud, Kamrul Hasan

AI总结:

本研究通过微调VideoMAE、ViViT和TimeSformer三种视频变换器模型,在BdSLW60和BdSLW401数据集上实现了高精度孟加拉手语识别,其中VideoMAE在帧率校正后的BdSLW60上达到95.5%准确率。

摘要:

手语识别(SLR)涉及从图像或视频中自动识别和分类手势,将其转换为文本或语音,以改善听障社区的可访问性。在孟加拉国,孟加拉手语(BdSL)是许多听障人士的主要交流方式。本研究在BdSLW60(arXiv:2402.08635)上微调了最先进的视频变换器架构——VideoMAE、ViViT和TimeSformer,BdSLW60是一个包含60个频繁手势的小规模BdSL数据集。我们将视频标准化为30 FPS,得到9,307个用户试用片段。为了评估可扩展性和鲁棒性,模型还在BdSLW401(arXiv:2503.02360)上进行了微调,这是一个包含401个手势类别的大规模数据集。此外,我们还在公开数据集(包括LSA64和WLASL)上进行了基准测试。应用了随机裁剪、水平翻转和短边缩放等数据增强技术以提高模型鲁棒性。为了在模型选择期间确保跨折的平衡评估,我们在训练集上采用了10折分层交叉验证,同时使用来自未见用户U4和U8的留出测试数据进行了独立于手语者的评估。结果表明,视频变换器模型显著优于传统的机器学习和深度学习方法。性能受数据集大小、视频质量、帧分布、帧率和模型架构等因素影响。在这些模型中,VideoMAE变体(MCG-NJU/videomae-base-finetuned-kinetics)在帧率校正后的BdSLW60数据集上达到了95.5%的最高准确率,在BdSLW401的正面手势上达到了81.04%——展示了可扩展且准确的BdSL识别的强大潜力。

英文摘要:

Sign Language Recognition (SLR) involves the automatic identification and classification of sign gestures from images or video, converting them into text or speech to improve accessibility for the hearing-impaired community. In Bangladesh, Bangla Sign Language (BdSL) serves as the primary mode of communication for many individuals with hearing impairments. This study fine-tunes state-of-the-art video transformer architectures -- VideoMAE, ViViT, and TimeSformer -- on BdSLW60 (arXiv:2402.08635), a small-scale BdSL dataset with 60 frequent signs. We standardized the videos to 30 FPS, resulting in 9,307 user trial clips. To evaluate scalability and robustness, the models were also fine-tuned on BdSLW401 (arXiv:2503.02360), a large-scale dataset with 401 sign classes. Additionally, we benchmark performance against public datasets, including LSA64 and WLASL. Data augmentation techniques such as random cropping, horizontal flipping, and short-side scaling were applied to improve model robustness. To ensure balanced evaluation across folds during model selection, we employed 10-fold stratified cross-validation on the training set, while signer-independent evaluation was carried out using held-out test data from unseen users U4 and U8. Results show that video transformer models significantly outperform traditional machine learning and deep learning approaches. Performance is influenced by factors such as dataset size, video quality, frame distribution, frame rate, and model architecture. Among the models, the VideoMAE variant (MCG-NJU/videomae-base-finetuned-kinetics) achieved the highest accuracies of 95.5% on the frame rate corrected BdSLW60 dataset and 81.04% on the front-facing signs of BdSLW401 -- demonstrating strong potential for scalable and accurate BdSL recognition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑