通过手语重建的生成对抗网络实现手语到语音语调转换
Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN
AI总结：
本文提出手语到语音语调转换任务，通过SignRecGAN和S2PFormer模型，在无需跨模态标注的情况下实现手语语调信息的直接整合，提升语音合成的情感表达。
摘要：
深度学习模型已提升了手语到文本的翻译能力，但将其转换为文本再合成语音的两阶段流程会丢失手语中丰富的非语言信息。为此，本文提出手语到语音语调转换任务，旨在捕捉手语中的全局语调特征并直接整合到合成语音中。主要挑战是手语与语音对齐需要专家知识，导致标注成本高且难以构建大规模平行语料库。为解决此问题，本文引入SignRecGAN，通过对抗学习和重建损失利用单模态数据进行可扩展训练。此外，提出S2PFormer模型，在保留现有TTS模型表现力的同时，实现将手语派生的语调注入合成语音中。大量实验表明，所提方法能合成忠实反映手语情感内容的语音，为更自然的手语交流开辟新可能。
英文摘要：
Deep learning models have improved sign language-to-text translation and made it easier for non-signers to understand signed messages. When the goal is spoken communication, a naive approach is to convert signed messages into text and then synthesize speech via Text-to-Speech (TTS). However, this two-stage pipeline inevitably treat text as a bottleneck representation, causing the loss of rich non-verbal information originally conveyed in the signing. To address this limitation, we propose a novel task, \emph{Sign-to-Speech Prosody Transfer}, which aims to capture the global prosodic nuances expressed in sign language and directly integrate them into synthesized speech. A major challenge is that aligning sign and speech requires expert knowledge, making annotation extremely costly and preventing the construction of large parallel corpora. To overcome this, we introduce \emph{SignRecGAN}, a scalable training framework that leverages unimodal datasets without cross-modal annotations through adversarial learning and reconstruction losses. Furthermore, we propose \emph{S2PFormer}, a new model architecture that preserves the expressive power of existing TTS models while enabling the injection of sign-derived prosody into the synthesized speech. Extensive experiments demonstrate that the proposed method can synthesize speech that faithfully reflects the emotional content of sign language, thereby opening new possibilities for more natural sign language communication. Our code will be available upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
