基于多提示者语料库的新型可解释、可泛化的提示语语音再同步模型
A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus
AI总结：
针对Cued Speech中唇手动作异步问题，提出基于唇音流三种统计度量和归一化的可解释、可泛化、与提示者无关的HPT预测模型，并构建首个含五位说话人的普通话CS语料库，实验性能优于现有方法。
摘要：
Cued Speech（CS，提示语语音）是一种多模态视觉编码系统，在语音层面将唇读与若干手部提示相结合，使口语对听障人士可见。以往研究通过针对英语和法语CS的依赖提示者的分段线性模型，解决了唇部与手部动作之间的异步问题。在本研究中，我们创新性地在唇音流上提出三种统计度量，以构建一个可解释且可泛化的模型来预测手部提前时间（HPT），并通过适当的归一化实现与提示者无关。特别是，我们构建了首个普通话CS语料库，包含来自五位说话人的标注视频，其中包括三位正常人和两位听障人士。因此，我们证明手部提前现象存在于普通话CS产生过程中，并且正常人与听障人士之间存在显著差异。大量实验表明，我们的模型优于基线以及以往最先进的方法。
英文摘要：
Cued Speech (CS) is a multi-modal visual coding system combining lip reading with several hand cues at the phonetic level to make the spoken language visible to the hearing impaired. Previous studies solved asynchronous problems between lip and hand movements by a cuer\footnote{The people who perform Cued Speech are called the cuer.}-dependent piecewise linear model for English and French CS. In this work, we innovatively propose three statistical measure on the lip stream to build an interpretable and generalizable model for predicting hand preceding time (HPT), which achieves cuer-independent by a proper normalization. Particularly, we build the first Mandarin CS corpus comprising annotated videos from five speakers including three normal and two hearing impaired individuals. Consequently, we show that the hand preceding phenomenon exists in Mandarin CS production with significant differences between normal and hearing impaired people. Extensive experiments demonstrate that our model outperforms the baseline and the previous state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
