基于多尺度时间对齐的领域自适应的跨手语迁移学习
Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment
- Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）
- The University（该大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究针对手语识别资源匮乏问题，采用带多尺度时间对齐的领域自适应方法TA3N，结合TRN模块，实验表明其在ASL识别上优于神经网络迁移学习，且RGB模式性能优于光流模式。
AI中文摘要：
手语是听障人士的重要交流方式，但超过100种不同手语的识别资源严重匮乏。为此，我们开展了基于迁移学习和领域自适应方法TA3N的手语识别研究，TA3N利用时间关系网络（TRN）模块对齐多尺度时间关系。研究发现，领域自适应相较于基于神经网络的迁移学习性能更优，尤其能提升美国手语（ASL）的识别效果；同时确定了对齐源域与目标域间短期时间特征的有效性。除使用RGB模式外，我们还针对手语样本的光流模式开展实验，最终确定多数情况下RGB模式的性能优于光流模式。本研究旨在改善以手语为主要交流方式的人群的可及性与交流条件。
英文摘要：
Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.
