UniCUE: 中国手语提示语音视频到语音生成统一识别与生成框架
UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation
AI总结：
UniCUE提出首个统一框架，直接从CS视频生成语音，无需中间文本，通过整合姿态感知视觉处理器、语义对齐池和VisioPhonetic适配器，提升语音生成精度。
摘要：
手语提示（CS）通过手写编码增强唇读，提供视觉音素提示，支持听障人士的精确语音感知。CS视频到语音生成（CSV2S）任务旨在将CS视频转换为可理解的语音信号。现有研究大多集中在CS识别（CSR）上，将视频内容转换为文本。因此，CSV2S的常见解决方案是将CSR与文本到语音（TTS）系统集成。然而，这种流程依赖于文本作为中间介质，可能导致语音与CS视频动态之间的时间错位和误差传播。相比之下，直接从CS视频生成音频语音（直接CSV2S）常面临固有的多模态复杂性和CS数据有限的问题。为解决这些挑战，我们提出了UniCUE，这是首个统一的CSV2S框架，直接从CS视频生成语音而不依赖中间文本。UniCUE的核心创新在于整合了理解任务（CSR），提供细粒度的CS视觉-语义提示以指导语音生成。具体而言，UniCUE集成了姿态感知的视觉处理器、一个语义对齐池，能够实现精确的视觉-语义映射，以及一个VisioPhonetic适配器，以在统一架构内连接理解和生成任务。为了支持这一框架，我们构建了UniCUE-HI，一个大规模的普通话CS数据集，包含11282个视频，来自14名提示者，包括听障和正常听力者。在该数据集上的广泛实验表明，UniCUE在多个评估指标上均达到最先进的性能。
英文摘要：
Cued Speech (CS) enhances lipreading via hand coding, offering visual phonemic cues that support precise speech perception for the hearing-impaired. The task of CS Video-to-Speech generation (CSV2S) aims to convert CS videos into intelligible speech signals. Most existing research focuses on CS Recognition (CSR), which transcribes video content into text. Consequently, a common solution for CSV2S is to integrate CSR with a text-to-speech (TTS) system. However, this pipeline relies on text as an intermediate medium, which may lead to error propagation and temporal misalignment between speech and CS video dynamics. In contrast, directly generating audio speech from CS video (direct CSV2S) often suffers from the inherent multimodal complexity and the limited availability of CS data. To address these challenges, we propose UniCUE, the first unified framework for CSV2S that directly generates speech from CS videos without relying on intermediate text. The core innovation of UniCUE lies in integrating an understanding task (CSR) that provides fine-grained CS visual-semantic cues to guide speech generation. Specifically, UniCUE incorporates a pose-aware visual processor, a semantic alignment pool that enables precise visual-semantic mapping, and a VisioPhonetic adapter to bridge the understanding and generation tasks within a unified architecture. To support this framework, we construct UniCUE-HI, a large-scale Mandarin CS dataset containing 11282 videos from 14 cuers, including both hearing-impaired and normal-hearing individuals. Extensive experiments on this dataset demonstrate that UniCUE achieves state-of-the-art performance across multiple evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
