跟随你的动作：一种具有轨迹引导的通用时间一致性肖像编辑框架
Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidance
AI总结：
提出一种通用肖像编辑框架FYM，利用扩散模型学习运动轨迹并引入动态重加权注意力机制，以解决说话头编辑中的时间一致性问题，优于现有方法。
摘要：
预训练的条件扩散模型在图像编辑领域已展现出显著潜力。然而，它们常常面临时间一致性的挑战，尤其是在说话头领域，面部表情的持续变化加剧了编辑的难度。这些问题源于对单个图像的独立编辑以及编辑过程中时间连续性的固有损失。在本文中，我们提出了“跟随你的动作”（Follow Your Motion, FYM），一个用于保持肖像编辑中时间一致性的通用框架。具体而言，给定由预训练的3D高斯泼溅模型渲染的肖像图像，我们首先开发了一个扩散模型，该模型从第一帧到每一后续帧，直观且固有地学习不同尺度和像素坐标下的运动轨迹变化。这种方法确保时间上不一致的编辑头像继承了渲染头像的运动信息。其次，为了在说话头编辑中保持细粒度的表情时间一致性，我们提出了一种动态重新加权的注意力机制。该机制为空间中的地标点分配更高的权重系数，并基于地标损失动态更新这些权重，从而实现更一致和精细的面部表情。大量实验表明，我们的方法在时间一致性方面优于现有方法，并可用于优化和补偿一系列应用中的时间不一致输出，例如文本驱动编辑、重新照明以及其他各种应用。
英文摘要：
Pre-trained conditional diffusion models have demonstrated remarkable potential in image editing. However, they often face challenges with temporal consistency, particularly in the talking head domain, where continuous changes in facial expressions intensify the level of difficulty. These issues stem from the independent editing of individual images and the inherent loss of temporal continuity during the editing process. In this paper, we introduce Follow Your Motion (FYM), a generic framework for maintaining temporal consistency in portrait editing. Specifically, given portrait images rendered by a pre-trained 3D Gaussian Splatting model, we first develop a diffusion model that intuitively and inherently learns motion trajectory changes at different scales and pixel coordinates, from the first frame to each subsequent frame. This approach ensures that temporally inconsistent edited avatars inherit the motion information from the rendered avatars. Secondly, to maintain fine-grained expression temporal consistency in talking head editing, we propose a dynamic re-weighted attention mechanism. This mechanism assigns higher weight coefficients to landmark points in space and dynamically updates these weights based on landmark loss, achieving more consistent and refined facial expressions. Extensive experiments demonstrate that our method outperforms existing approaches in terms of temporal consistency and can be used to optimize and compensate for temporally inconsistent outputs in a range of applications, such as text-driven editing, relighting, and various other applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
