LawDNet:通过局部仿射翘曲变形增强音频驱动的唇部合成
LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation
AI总结:
针对音频驱动唇部合成中姿态多样性不足与时间连贯性差的问题,提出LawDNet架构,通过局部仿射翘曲变形机制建模唇部运动,结合双流判别器与人脸归一化技术,提升了合成的鲁棒性与动态性。
摘要:
在照片级虚拟形象生成领域,音频驱动的唇部动作合成的逼真度对真实虚拟交互至关重要。现有方法面临两大关键挑战:生成的唇部姿态多样性有限导致缺乏生动性,以及时间连贯性差造成明显的变形动作。为解决这些问题,我们提出LawDNet,一种通过局部仿射翘曲变形机制增强唇部合成的新型深度学习架构。该机制通过可控的非线性翘曲场对响应音频输入的复杂唇部运动进行建模。这些翘曲场由聚焦于深度特征图中抽象关键点的局部仿射变换组成,为网络中的特征翘曲提供了一种新颖的通用范式。此外,LawDNet引入了双流判别器以提升帧间连续性,并采用人脸归一化技术处理姿态和场景变化。大量评估表明,与现有方法相比,LawDNet在鲁棒性和唇部运动动态性方面表现更优。本文提出的进展,包括方法、训练数据、源代码和预训练模型,将向研究界开放。
英文摘要:
In the domain of photorealistic avatar generation, the fidelity of audio-driven lip motion synthesis is essential for realistic virtual interactions. Existing methods face two key challenges: a lack of vivacity due to limited diversity in generated lip poses and noticeable anamorphose motions caused by poor temporal coherence. To address these issues, we propose LawDNet, a novel deep-learning architecture enhancing lip synthesis through a Local Affine Warping Deformation mechanism. This mechanism models the intricate lip movements in response to the audio input by controllable non-linear warping fields. These fields consist of local affine transformations focused on abstract keypoints within deep feature maps, offering a novel universal paradigm for feature warping in networks. Additionally, LawDNet incorporates a dual-stream discriminator for improved frame-to-frame continuity and employs face normalization techniques to handle pose and scene variations. Extensive evaluations demonstrate LawDNet's superior robustness and lip movement dynamism performance compared to previous methods. The advancements presented in this paper, including the methodologies, training data, source codes, and pre-trained models, will be made accessible to the research community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。