arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.14698cs.CVcs.LGcs.NIcs.SI
收藏

面部标志预测及其在Metaverse中的应用

Facial Landmark Predictions with Applications to Metaverse

Nanyang Technological University(南洋理工大学)

Qiao Han, Jun Zhao, Kwok-Yan Lam

摘要:

这项研究旨在通过添加从野外视频中学习的嘴唇动画来使虚拟世界角色更加真实。为了实现这一点,我们的方法是扩展Tacotron 2文本到语音合成器,以在一个通道中生成嘴唇运动和梅尔频谱图。编码器“与”门层权重在LJ Speech 1.1数据集上进行预训练,而解码器在从LRS 3数据集提取的93个TED演讲视频片段上进行重新训练。我们的新解码器预测位移在20唇地标位置随着时间的推移,使用标签自动提取的OpenFace 2.0地标预测。培训在7小时内完成,使用不到5分钟的视频。我们对前/后网络和预训练的编码器权重进行了消融研究,以证明音频和视觉语音数据之间的迁移学习的有效性。

英文摘要:

This research aims to make metaverse characters more realistic by adding lip animations learnt from videos in the wild. To achieve this, our approach is to extend Tacotron 2 text-to-speech synthesizer to generate lip movements together with mel spectrogram in one pass. The encoder and gate layer weights are pre-trained on LJ Speech 1.1 data set while the decoder is retrained on 93 clips of TED talk videos extracted from LRS 3 data set. Our novel decoder predicts displacement in 20 lip landmark positions across time, using labels automatically extracted by OpenFace 2.0 landmark predictor. Training converged in 7 hours using less than 5 minutes of video. We conducted ablation study for Pre/Post-Net and pre-trained encoder weights to demonstrate the effectiveness of transfer learning between audio and visual speech data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑