低资源场景下尼泊尔口语词汇到情感条件手语虚拟人物的多模态翻译
Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars
AI总结：
提出NEST-V1框架，通过共享声学编码器实现语音识别与情感分类，生成情感条件尼泊尔手语虚拟人物，在低资源场景下验证了技术可行性。
摘要：
整合情感表达的手语通信系统仍未得到充分探索，尤其是对于低资源语言。本初步研究提出了NEST-V1（尼泊尔情感与语音转换器-版本1），一个概念验证的多模态框架，展示了从语音输入生成情感条件的尼泊尔手语虚拟人物的可行性。作为初步调查，我们聚焦于四个常见尼泊尔词汇（“谢谢”、“你好”、“房子”、“我”）在三种情感状态（快乐、中性、悲伤）下，以验证我们的核心技术方法。我们的轻量级架构采用共享声学编码器同时进行自动语音识别和情感分类，在来自50名说话者的600个标记音频样本数据集上实现了81.1%的ASR准确率和79.21%的情感识别准确率。与分离模型架构相比，该系统实现了37%的参数效率，同时保持了仅22.1M参数的轻量级足迹，适合边缘部署。这项初步工作为低资源环境下的情感感知手语翻译奠定了技术基础，并提供了一个可扩展的框架，以便未来扩展到更大的词汇量和更多样化的情感表达。我们的初步结果表明，为听障社区实现实时、情感表达的手语通信系统是可行的，并在后续开发阶段有明确的改进路径。
英文摘要：
Sign language communication systems, that integrate emotional expression remain underexplored, particularly for low-resource languages. This pilot study presents NEST-V1 (Nepali Emotion and Speech Transformer - Version 1), a proof-of-concept multimodal framework that demonstrates the feasibility of generating emotion-conditioned Nepali Sign Language avatars from spoken input. As a preliminary investigation, we focus on four common Nepali words ("thank you", "hello", "house", "me") across three emotional states (happy, neutral, sad) to validate our core technical approach. Our lightweight architecture employs a shared acoustic encoder for simultaneous Automatic Speech Recognition and emotion classification, achieving 81.1% ASR accuracy and 79.21% emotion recognition accuracy on a dataset of 600 labeled audio samples from 50 speakers. The system demonstrates 37% parameter efficiency compared to separate model architectures while maintaining a lightweight footprint with only 22.1M parameters suitable for edge deployment. This pilot work establishes the technical foundation for emotion-aware sign language translation in low-resource settings and provides a scalable framework for future expansion to larger vocabularies and more diverse emotional expressions. Our preliminary results indicate the viability of real-time, emotionally expressive sign language communication systems for the hearing-impaired community, with clear pathways for enhancement in subsequent development phases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
