基于空间条件扩散的一致性人物图像与视频生成
Consistent Human Image and Video Generation with Spatially Conditioned Diffusion
AI总结：
本文提出一种基于空间条件扩散的一致性人物图像与视频生成方法，通过将任务构建为空间条件修复问题，在统一去噪网络内利用因果特征交互框架，有效缓解域间隙并保持外观一致性，实现强泛化能力。
摘要：
一致性以人为中心的图像与视频合成旨在生成具有新姿态的图像或视频，同时保持与给定参考图像的外观一致性，这对于低成本的视觉内容创作至关重要。近期基于扩散模型的进展通常依赖分离的网络来进行参考外观特征提取和目标视觉生成，导致参考与目标之间存在不一致的域间隙。在本文中，我们将该任务构建为空间条件修复问题，其中目标图像被修复以保持与参考图像的外观一致性。该方法使参考特征能够在统一的去噪网络内指导生成符合姿态的目标，从而缓解域间隙。此外，为了更好地保持参考外观信息，我们施加了因果特征交互框架，其中参考特征只能从自身查询，而目标特征可以从参考和目标两者查询外观信息。为了进一步提高计算效率和灵活性，在实际实现中，我们将空间条件生成过程分解为两个阶段：参考外观提取和条件目标生成。两个阶段共享单个去噪网络，且交互仅限于自注意力层。所提出的方法确保了对生成人物图像和视频外观的灵活控制。通过在人类视频数据上微调现有的基础扩散模型，我们的方法展示了对未见人物身份和姿态的强泛化能力，无需额外的逐实例微调。实验结果验证了我们方法的有效性，与现有方法相比，在一致性人物图像与视频合成方面表现出有竞争力的性能。
英文摘要：
Consistent human-centric image and video synthesis aims to generate images or videos with new poses while preserving appearance consistency with a given reference image, which is crucial for low-cost visual content creation. Recent advances based on diffusion models typically rely on separate networks for reference appearance feature extraction and target visual generation, leading to inconsistent domain gaps between references and targets. In this paper, we frame the task as a spatially-conditioned inpainting problem, where the target image is inpainted to maintain appearance consistency with the reference. This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network, thereby mitigating domain gaps. Additionally, to better maintain the reference appearance information, we impose a causal feature interaction framework, in which reference features can only query from themselves, while target features can query appearance information from both the reference and the target. To further enhance computational efficiency and flexibility, in practical implementation, we decompose the spatially-conditioned generation process into two stages: reference appearance extraction and conditioned target generation. Both stages share a single denoising network, with interactions restricted to self-attention layers. This proposed method ensures flexible control over the appearance of generated human images and videos. By fine-tuning existing base diffusion models on human video data, our method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning. Experimental results validate the effectiveness of our approach, showing competitive performance compared to existing methods for consistent human image and video synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
