你们想跳舞吗：面向多人的零样本组合式人类舞蹈生成
Do You Guys Want to Dance: Zero-Shot Compositional Human Dance Generation with Multiple Persons
AI总结：
针对多人复杂背景下人类舞蹈生成泛化不足的问题，提出cHDG基准与零样本框架MultiDance-Zero，通过姿态感知反演、组合增强和一致性引导采样提升多人视频生成质量。
摘要：
人类舞蹈生成（HDG）旨在根据图像和驱动姿态序列合成逼真视频。尽管取得了巨大成功，现有方法仍局限于生成具有特定背景的单人视频，而其在包含多人和复杂背景的真实场景中的泛化性仍不清楚。为系统衡量HDG模型的泛化性，我们提出了组合式人类舞蹈生成（cHDG）这一新任务、数据集和评估协议。在cHDG上评估最先进方法时，我们通过实验发现它们无法泛化到真实场景。为解决该问题，我们提出了一个名为MultiDance-Zero的新型零样本框架，能够在精确遵循驱动姿态的同时，合成与任意多人及背景一致的视频。具体而言，与直接使用DDIM或null-text inversion不同，我们首先提出一种姿态感知反演方法，以获得噪声潜码和初始化文本嵌入，从而能够准确重建组合后的参考图像。由于直接由它们生成视频会导致严重的外观不一致，我们提出了一种组合增强策略来生成增强图像，并利用这些图像优化一组可泛化的文本嵌入。此外，我们精心设计了一致性引导采样，促使每个反向步骤中估计的干净图像的背景和关键点接近参考图像，进一步提升生成视频的时间一致性。大量定性和定量结果证明了本方法的有效性和优越性。
英文摘要：
Human dance generation (HDG) aims to synthesize realistic videos from images and sequences of driving poses. Despite great success, existing methods are limited to generating videos of a single person with specific backgrounds, while the generalizability for real-world scenarios with multiple persons and complex backgrounds remains unclear. To systematically measure the generalizability of HDG models, we introduce a new task, dataset, and evaluation protocol of compositional human dance generation (cHDG). Evaluating the state-of-the-art methods on cHDG, we empirically find that they fail to generalize to real-world scenarios. To tackle the issue, we propose a novel zero-shot framework, dubbed MultiDance-Zero, that can synthesize videos consistent with arbitrary multiple persons and background while precisely following the driving poses. Specifically, in contrast to straightforward DDIM or null-text inversion, we first present a pose-aware inversion method to obtain the noisy latent code and initialization text embeddings, which can accurately reconstruct the composed reference image. Since directly generating videos from them will lead to severe appearance inconsistency, we propose a compositional augmentation strategy to generate augmented images and utilize them to optimize a set of generalizable text embeddings. In addition, consistency-guided sampling is elaborated to encourage the background and keypoints of the estimated clean image at each reverse step to be close to those of the reference image, further improving the temporal consistency of generated videos. Extensive qualitative and quantitative results demonstrate the effectiveness and superiority of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
