通往无障碍沟通之桥：基于扩散模型的Gloss提示细粒度手语手势生成
Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech Gesture Generation with Diffusion Model
AI总结：
本文提出GlossDiff框架，通过引入Gloss桥接指令和音频驱动节奏模块，基于扩散模型同时生成细粒度且异步对齐的手势与唇部动作，并发布了首个中文CS数据集，性能优于现有SOTA方法。
摘要：
手语（Cued Speech, CS）是一种先进的视觉语音编码系统，它将唇读与手势编码相结合，使听障人士能够高效沟通。CS视频生成旨在从音频或文本输入中生成CS的特定唇部和手势动作。主要挑战在于：在CS数据有限的情况下，我们需要同时生成细粒度的手部和手指动作以及唇部动作，同时这两类动作需要异步对齐。现有的CS生成方法较为脆弱，由于基于模板的统计模型和精心设计的手工预处理以适应模型，容易导致性能不佳。因此，我们提出了一种新颖的基于Gloss提示的扩散CS手势生成框架（称为GlossDiff）。具体而言，为了将额外的语言规则知识融入模型，我们首先引入了一种称为Gloss的桥接指令，它是一种自动生成的描述性文本，用于在口语和CS手势之间建立直接且更精细的语义联系。此外，我们首次提出节奏是CS中重要的副语言特征，可提高沟通效率。因此，我们提出了一种新颖的音频驱动节奏模块（ARM）来学习与音频语音匹配的节奏。此外，在这项工作中，我们设计、录制并发布了首个包含四位CS演示者的中文CS数据集。大量实验表明，我们的方法在定量和定性上均优于当前最先进（SOTA）方法。我们在https://glossdiff.github.io/上发布了代码和数据。
英文摘要：
Cued Speech (CS) is an advanced visual phonetic encoding system that integrates lip reading with hand codings, enabling people with hearing impairments to communicate efficiently. CS video generation aims to produce specific lip and gesture movements of CS from audio or text inputs. The main challenge is that given limited CS data, we strive to simultaneously generate fine-grained hand and finger movements, as well as lip movements, meanwhile the two kinds of movements need to be asynchronously aligned. Existing CS generation methods are fragile and prone to poor performance due to template-based statistical models and careful hand-crafted pre-processing to fit the models. Therefore, we propose a novel Gloss-prompted Diffusion-based CS Gesture generation framework (called GlossDiff). Specifically, to integrate additional linguistic rules knowledge into the model. we first introduce a bridging instruction called \textbf{Gloss}, which is an automatically generated descriptive text to establish a direct and more delicate semantic connection between spoken language and CS gestures. Moreover, we first suggest rhythm is an important paralinguistic feature for CS to improve the communication efficacy. Therefore, we propose a novel Audio-driven Rhythmic Module (ARM) to learn rhythm that matches audio speech. Moreover, in this work, we design, record, and publish the first Chinese CS dataset with four CS cuers. Extensive experiments demonstrate that our method quantitatively and qualitatively outperforms current state-of-the-art (SOTA) methods. We release the code and data at https://glossdiff.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
