LokiTalk：学习细粒度且可泛化的对应关系以增强基于NeRF的说话头部合成
LokiTalk: Learning Fine-Grained and Generalizable Correspondences to Enhance NeRF-based Talking Head Synthesis
AI总结：
提出LokiTalk框架，通过Region-Specific Deformation Fields和ID-Aware Knowledge Transfer学习细粒度且可泛化的对应关系，有效减少基于NeRF的说话头部合成中的视觉伪影并提升训练效率。
摘要：
自从引入Neural Radiance Fields (NeRF)以来，说话头部合成取得了显著进展，但视觉伪影和高昂的训练成本仍是大规模商业应用的主要障碍。我们提出，识别并建立驱动信号与生成结果之间细粒度且可泛化的对应关系，可以同时解决这两个问题。在此我们提出LokiTalk，这是一个旨在通过栩栩如生的面部动态和提升的训练效率来增强基于NeRF的说话头部的新颖框架。为实现细粒度对应，我们引入了Region-Specific Deformation Fields，将整体肖像运动分解为唇部运动、眨眼、头部姿态和躯干运动。通过两个级联变形场对驱动信号及其关联区域进行分层建模，我们显著提高了动态精度并最小化了合成伪影。此外，我们提出了ID-Aware Knowledge Transfer，这是一个即插即用的模块，从多身份视频中学习可泛化的动态和静态对应关系，同时提取特定身份的动态和静态特征以精炼个体角色的描绘。综合评估表明，与以往方法相比，LokiTalk提供了更优越的高保真结果和训练效率。代码将在论文被接受后发布。
英文摘要：
Despite significant progress in talking head synthesis since the introduction of Neural Radiance Fields (NeRF), visual artifacts and high training costs persist as major obstacles to large-scale commercial adoption. We propose that identifying and establishing fine-grained and generalizable correspondences between driving signals and generated results can simultaneously resolve both problems. Here we present LokiTalk, a novel framework designed to enhance NeRF-based talking heads with lifelike facial dynamics and improved training efficiency. To achieve fine-grained correspondences, we introduce Region-Specific Deformation Fields, which decompose the overall portrait motion into lip movements, eye blinking, head pose, and torso movements. By hierarchically modeling the driving signals and their associated regions through two cascaded deformation fields, we significantly improve dynamic accuracy and minimize synthetic artifacts. Furthermore, we propose ID-Aware Knowledge Transfer, a plug-and-play module that learns generalizable dynamic and static correspondences from multi-identity videos, while simultaneously extracting ID-specific dynamic and static features to refine the depiction of individual characters. Comprehensive evaluations demonstrate that LokiTalk delivers superior high-fidelity results and training efficiency compared to previous methods. The code will be released upon acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
