利用语言先验从视频重建手语虚拟化身
Reconstructing Signing Avatars From Video Using Linguistic Priors
AI总结：
本文提出SGNify，利用适用于手语的语言先验约束3D手部姿态，从野外单目视频自动重建富有表现力的手语虚拟化身，并在可懂度与自然度上超越现有方法。
摘要：
手语（SL）是全球7000万听障人士的主要交流方式。孤立手语词汇的视频词典是核心的手语学习工具。用3D虚拟化身替代这些视频有助于学习，并支持AR/VR应用，从而改善人们获取技术和在线媒体的途径。然而，很少有研究尝试从手语视频估计富有表现力的3D虚拟化身；遮挡、噪声和运动模糊使这项任务十分困难。我们通过引入新颖的语言先验来解决这一问题，这些先验普遍适用于手语，并对3D手部姿态提供约束，有助于消除孤立手语中的歧义。我们的方法SGNify能够完全自动地从野外单目手语视频中捕捉细粒度的手部姿态、面部表情和身体运动。我们使用商用动作捕捉系统计算与单目视频同步的3D虚拟化身，对SGNify进行定量评估。在手语视频上，SGNify优于最先进的3D身体姿态与形状估计方法。一项感知研究表明，SGNify的3D重建结果比以往方法的结果明显更易理解、更自然，并且与源视频相当。代码和数据可在sgnify.is.tue.mpg.de获取。
英文摘要：
Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video; occlusion, noise, and motion blur make this task difficult. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose- and shape-estimation methods on SL videos. A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos. Code and data are available at $\href{http://sgnify.is.tue.mpg.de}{\text{sgnify.is.tue.mpg.de}}$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
