协同运动与外观：用于说话头视频生成的多尺度补偿码本
Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation
AI总结：
针对说话头视频生成中姿态与细节难以兼顾的问题，提出联合学习运动与外观码本并进行多尺度补偿，以提升生成质量。
摘要：
说话头视频生成旨在从源图像中保留人物身份，并从驱动视频中获取运动信息，从而生成逼真的说话头视频。尽管该领域已取得显著进展，但同时生成具有精确姿态和细粒度面部细节的视频仍然是一个具有挑战性的关键问题。本质上，面部运动通常高度复杂，难以精确建模，并且由于动态姿态变化，单次拍摄的源人脸图像在生成过程中无法提供足够的外观指导。为解决这一问题，我们提出联合学习运动码本和外观码本，并进行多尺度码本补偿，以有效细化面部运动条件和外观特征，用于说话人脸图像解码。具体而言，所设计的运动码本和外观码本在统一框架中同时学习，以存储具有代表性的全局面部运动流和外观模式。然后，我们提出一种新颖的多尺度运动与外观补偿模块，该模块利用基于Transformer的码本检索策略，从两个码本中查询互补信息，以实现联合运动与外观补偿。整个过程在不同尺度上产生更灵活的运动流和更少失真的外观特征，从而构建出高质量的说话头视频生成框架。在各种基准上的大量实验验证了我们方法的有效性，并在定性和定量角度均展示了相较于最先进竞争对手的优越生成结果。
英文摘要：
Talking head video generation aims to generate a realistic talking head video that preserves the person's identity from a source image and the motion from a driving video. Despite the promising progress made in the field, it remains a challenging and critical problem to generate videos with accurate poses and fine-grained facial details simultaneously. Essentially, facial motion is often highly complex to model precisely, and the one-shot source face image cannot provide sufficient appearance guidance during generation due to dynamic pose changes. To tackle the problem, we propose to jointly learn motion and appearance codebooks and perform multi-scale codebook compensation to effectively refine both the facial motion conditions and appearance features for talking face image decoding. Specifically, the designed multi-scale motion and appearance codebooks are learned simultaneously in a unified framework to store representative global facial motion flow and appearance patterns. Then, we present a novel multi-scale motion and appearance compensation module, which utilizes a transformer-based codebook retrieval strategy to query complementary information from the two codebooks for joint motion and appearance compensation. The entire process produces motion flows of greater flexibility and appearance features with fewer distortions across different scales, resulting in a high-quality talking head video generation framework. Extensive experiments on various benchmarks validate the effectiveness of our approach and demonstrate superior generation results from both qualitative and quantitative perspectives when compared to state-of-the-art competitors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
