arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.06363cs.CV
收藏

用于与文本无关的说话人识别的三维唇动网络

Three-Dimensional Lip Motion Network for Text-Independent Speaker Recognition

College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Tianjin Key Laboratory of Cognitive Computing and Application(天津市认知计算与应用重点实验室);Institute of Linguistics, Chinese Academy of Social Sciences(中国社会科学院语言研究所);Shenzhen Research Institute of Big Data, the chinese university of hong kong shenzhen(香港中文大学(深圳)深圳大数据研究院)

Jianrong Wang, Tong Wu, Shanyu Wang, Mei Yu, Qiang Fang, Ju Zhang, Li Liu

摘要:

唇动特征反映了说话人的行为特征,可以作为一种新的生物特征用于说话人识别。在文献中,大量的工作使用二维(2D)唇图像识别说话人在文本相关的背景。然而,2D唇容易受到各种面取向的影响。为此,在这项工作中,我们提出了一种新的端到端的3D唇运动网络(3LMNet),利用双音级3D唇运动(S3 DLM)识别说话人在文本无关和文本相关的上下文中。提出了一种新的区域反馈模块(RFM),以获得不同唇区的注意力。此外,唇运动的先验知识进行了调查,以补充RFM,其中的地标级和帧级的特征合并,形成一个更好的特征表示。此外,我们提出了两种方法,即,坐标变换和人脸姿态校正来预处理LSD-AV数据集,该数据集包含68个说话人和每个说话人146个句子。对该数据集的评估结果表明,我们提出的3LMNet优于基线模型,即,LSTM,VGG-16和ResNet-34,使用2D唇部图像以及3D面部,性能优于最先进的技术。该工作的代码发布在https://github.com/wutong18/Three-Dimensional-Lip- Motion-Network-for-Text-Independent-Speaker-Recognition上。

英文摘要:

Lip motion reflects behavior characteristics of speakers, and thus can be used as a new kind of biometrics in speaker recognition. In the literature, lots of works used two-dimensional (2D) lip images to recognize speaker in a textdependent context. However, 2D lip easily suffers from various face orientations. To this end, in this work, we present a novel end-to-end 3D lip motion Network (3LMNet) by utilizing the sentence-level 3D lip motion (S3DLM) to recognize speakers in both the text-independent and text-dependent contexts. A new regional feedback module (RFM) is proposed to obtain attentions in different lip regions. Besides, prior knowledge of lip motion is investigated to complement RFM, where landmark-level and frame-level features are merged to form a better feature representation. Moreover, we present two methods, i.e., coordinate transformation and face posture correction to pre-process the LSD-AV dataset, which contains 68 speakers and 146 sentences per speaker. The evaluation results on this dataset demonstrate that our proposed 3LMNet is superior to the baseline models, i.e., LSTM, VGG-16 and ResNet-34, and outperforms the state-of-the-art using 2D lip image as well as the 3D face. The code of this work is released at https://github.com/wutong18/Three-Dimensional-Lip- Motion-Network-for-Text-Independent-Speaker-Recognition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑