面向目标的视频扩散模型
Target-Aware Video Diffusion Models
AI总结：
本文提出一种面向目标的视频扩散模型，通过引入目标掩码和特殊标记提升视频生成中演员与目标的互动准确性，并在两个下游任务中验证其有效性。
摘要：
我们提出了一种面向目标的视频扩散模型，该模型能够从输入图像生成视频，在其中演员与指定的目标进行互动并执行期望的动作。目标由分割掩码定义，动作则通过文本提示描述。我们的核心动机是将目标意识融入视频生成中，使演员能够对指定对象执行定向动作。这使视频扩散模型能够充当运动规划器，通过利用大规模视频生成模型的先验知识，生成合理的人-物交互预测。我们通过扩展基线模型，将目标掩码作为额外输入来构建我们的目标意识模型。为了强制目标意识，我们引入了一个特殊标记，该标记在文本提示中编码目标的空间信息。然后我们使用额外的交叉注意力损失对模型进行微调，该损失将与该标记相关的交叉注意力图与输入目标掩码对齐。为进一步提高性能，我们选择性地将此损失应用于最语义相关的注意力区域和Transformer块。实验结果表明，我们的目标意识模型在生成演员与指定目标准确互动的视频方面优于现有解决方案。我们进一步在两个下游应用中展示了其有效性：零样本3D HOI运动合成（具有物理合理性）和长期视频内容创作。
英文摘要：
We present a target-aware video diffusion model that generates videos from an input image, in which an actor interacts with a specified target while performing a desired action. The target is defined by a segmentation mask, and the action is described through a text prompt. Our key motivation is to incorporate target awareness into video generation, enabling actors to perform directed actions on designated objects. This enables video diffusion models to act as motion planners, producing plausible predictions of human-object interactions by leveraging the priors of large-scale video generative models. We build our target-aware model by extending a baseline model to incorporate the target mask as an additional input. To enforce target awareness, we introduce a special token that encodes the target's spatial information within the text prompt. We then fine-tune the model with our curated dataset using an additional cross-attention loss that aligns the cross-attention maps associated with this token with the input target mask. To further improve performance, we selectively apply this loss to the most semantically relevant attention regions and transformer blocks. Experimental results show that our target-aware model outperforms existing solutions in generating videos where actors interact accurately with the specified targets. We further demonstrate its efficacy in two downstream applications: zero-shot 3D HOI motion synthesis with physical plausibility and long-term video content creation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
