R2-Talker：采用哈希网格关键点编码与渐进式多层条件化的逼真实时说话头部合成
R2-Talker: Realistic Real-Time Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioning
AI总结：
本文提出 R2-Talker，利用多分辨率哈希网格无损编码面部关键点并在 NeRF 各 MLP 层渐进融合条件特征，实现高真实感、实时且泛化性更好的说话头部合成。
摘要：
动态 NeRF 最近在三维说话肖像合成领域受到越来越多关注。尽管渲染速度和视觉质量已有提升，但在增强效率和效果方面仍存在挑战。我们提出 R2-Talker，这是一个高效且有效的框架，可实现逼真的实时说话头部合成。具体而言，我们利用多分辨率哈希网格，提出一种将面部关键点编码为条件特征的新方法。该方法将关键点结构无损编码为条件特征，并通过把任意关键点映射到统一特征空间，解耦输入多样性与条件空间。我们还在 NeRF 渲染流程中提出渐进式多层条件化方案，以实现有效的条件特征融合。与最先进工作相比，大量实验表明我们的新方法具有以下优势：1）无损输入编码能够获得更精确的特征，从而带来更优视觉质量；输入与条件空间的解耦提升了泛化能力。2）在每个 MLP 层融合条件特征与 MLP 输出，可增强条件作用，带来更准确的唇部合成和更好的视觉质量。3）该方法以紧凑结构组织条件特征融合，显著提高了计算效率。
英文摘要：
Dynamic NeRFs have recently garnered growing attention for 3D talking portrait synthesis. Despite advances in rendering speed and visual quality, challenges persist in enhancing efficiency and effectiveness. We present R2-Talker, an efficient and effective framework enabling realistic real-time talking head synthesis. Specifically, using multi-resolution hash grids, we introduce a novel approach for encoding facial landmarks as conditional features. This approach losslessly encodes landmark structures as conditional features, decoupling input diversity, and conditional spaces by mapping arbitrary landmarks to a unified feature space. We further propose a scheme of progressive multilayer conditioning in the NeRF rendering pipeline for effective conditional feature fusion. Our new approach has the following advantages as demonstrated by extensive experiments compared with the state-of-the-art works: 1) The lossless input encoding enables acquiring more precise features, yielding superior visual quality. The decoupling of inputs and conditional spaces improves generalizability. 2) The fusing of conditional features and MLP outputs at each MLP layer enhances conditional impact, resulting in more accurate lip synthesis and better visual quality. 3) It compactly structures the fusion of conditional features, significantly enhancing computational efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
