PortraitDirector: 一种用于可控和实时面部重演的分层解耦框架
PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment
AI总结：
本文提出PortraitDirector框架，通过分层解耦策略实现高保真可控的面部重演，采用空间层和语义层解耦面部运动，并通过优化实现20FPS的实时重演。
摘要：
现有面部重演方法在表达性和细粒度可控性之间存在权衡问题。整体面部重演模型通常牺牲细粒度控制以换取表达性，而专门设计的控制方法则在保真度和鲁棒解耦方面面临挑战。本文提出了一种替代的组合视角，将面部运动视为单一信号。我们引入PortraitDirector框架，将面部重演建模为分层组合任务，实现高保真和可控的结果。我们采用分层运动解耦和组合策略，将面部运动分解为空间层（用于物理运动）和语义层（用于情感内容）。空间层包括：（i）全局头部姿态，通过专用表示和注入路径管理；（ii）空间分离的局部面部表情，从裁剪的面部区域中提取，并通过利用信息瓶颈的Emotion-Filtering模块去除情感线索。语义层包含派生的全局情感。解耦的组件随后重新组合成表达性运动潜在特征。此外，我们通过一系列优化工程使框架实现实时性能，包括扩散蒸馏、因果注意力和VAE加速。PortraitDirector在单个5090 GPU上实现20FPS的实时重演，以800ms端到端延迟生成512x512的高保真可控面部重演。
英文摘要：
Existing facial reenactment methods struggle with a trade-off between expressiveness and fine-grained controllability. Holistic facial reenactment models often sacrifice granular control for expressiveness, while methods designed for control may struggle with fidelity and robust disentanglement. Instead of treating facial motion as a monolithic signal, we explore an alternative compositional perspective. In this paper, we introduce PortraitDirector, a novel framework that formulates face reenactment as a hierarchical composition task, achieving high-fidelity and controllable results. We employ a Hierarchical Motion Disentanglement and Composition strategy, deconstructing facial motion into a Spatial Layer for physical movements and a Semantic Layer for emotional content. The Spatial Layer comprises: (i) global head pose, managed via a dedicated representation and injection pathway; (ii) spatially separated local facial expressions, distilled from cropped facial regions and purged of emotional cues via Emotion-Filtering Module leveraging an information bottleneck. The Semantic Layer contains a derived global emotion. The disentangled components are then recomposed into an expressive motion latent. Furthermore, we engineer the framework for real-time performance through a suite of optimizations, including diffusion distillation, causal attention and VAE acceleration. PortraitDirector achieves streaming, high-fidelity, controllable 512 x 512 face reenactment at 20 FPS with a end-to-end 800 ms latency on a single 5090 GPU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
