Three-Dimensional Lip Motion Network for Text-Independent Speaker Recognition
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Tianjin Key Laboratory of Cognitive Computing and Application(天津市认知计算与应用重点实验室);Institute of Linguistics, Chinese Academy of Social Sciences(中国社会科学院语言研究所);Shenzhen Research Institute of Big Data, the chinese university of hong kong shenzhen(香港中文大学(深圳)深圳大数据研究院)
Jianrong Wang, Tong Wu, Shanyu Wang, Mei Yu, Qiang Fang, Ju Zhang, Li Liu
Lip motion reflects behavior characteristics of speakers, and thus can be used as a new kind of biometrics in speaker recognition. In the literature, lots of works used two-dimensional (2D) lip images to recognize speaker in a textdependent context. However, 2D lip easily suffers from various face orientations. To this end, in this work, we present a novel end-to-end 3D lip motion Network (3LMNet) by utilizing the sentence-level 3D lip motion (S3DLM) to recognize speakers in both the text-independent and text-dependent contexts. A new regional feedback module (RFM) is proposed to obtain attentions in different lip regions. Besides, prior knowledge of lip motion is investigated to complement RFM, where landmark-level and frame-level features are merged to form a better feature representation. Moreover, we present two methods, i.e., coordinate transformation and face posture correction to pre-process the LSD-AV dataset, which contains 68 speakers and 146 sentences per speaker. The evaluation results on this dataset demonstrate that our proposed 3LMNet is superior to the baseline models, i.e., LSTM, VGG-16 and ResNet-34, and outperforms the state-of-the-art using 2D lip image as well as the 3D face. The code of this work is released at https://github.com/wutong18/Three-Dimensional-Lip- Motion-Network-for-Text-Independent-Speaker-Recognition.
Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization
School of Life Science and Technology, University of Electronic Science and Technology of China(电子科技大学生命科学与技术学院);Defense Innovation Institute, Academy of Military Sciences(军事科学院国防创新研究院);Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心);High-tech Institute of Qingzhou(青州高新技术研究院)
TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos
University of Edinburgh(爱丁堡大学);University of Science and Technology of China(中国科学技术大学);Queen Margaret University(玛格丽特女王大学);Articulate Instruments Ltd.(Articulate仪器有限公司)
我们提出了舌头和嘴唇语料库(TaL),音频,超声舌头成像和嘴唇视频的多扬声器语料库。TaL由两部分组成:TaL 1是一个专业的语音人才,英语为母语的男性的一套六个录音会话; TaL 80是一个没有语音人才经验的81个英语为母语的录音会话。总体而言,语料库包含24小时的并行超声、视频和音频数据,其中大约13.5小时是语音。本文介绍了语料库,并提出了基准测试结果的任务,语音识别,语音合成(发音声学映射),和自动同步的超声音频。TaL语料库在CC BY-NC 4.0许可下公开可用。
Speaker Recognition in Realistic Scenario Using Multimodal Data
University of Engineering and Technology Taxila(塔克西拉工程技术大学);Swarm Robotics Lab NCRA(国家机器人与自动化研究所群体机器人实验室);Deutsches Elektronen-Synchrotron DESY(德国电子同步加速器研究所)