VITON-DiT：基于扩散Transformer从人类舞蹈视频中学习真实场景视频试穿
VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers
AI总结：
本文提出首个基于扩散Transformer的视频试穿框架VITON-DiT，通过融合服装特征与时空去噪DiT及身份保持ControlNet，仅利用非配对舞蹈视频训练，实现真实场景复杂姿势下的长视频试穿生成。
摘要：
视频试穿因其巨大的现实世界潜力而成为一个有前景的领域。以往的工作仅限于将产品服装图像转移到具有简单姿势和背景的人物视频上，而在随意拍摄的视频上表现不佳。最近，Sora揭示了扩散Transformer（DiT）在生成具有真实世界场景的逼真视频方面的可扩展性。受此启发，我们探索并提出了首个基于DiT的视频试穿框架，用于实际的真实场景应用，命名为VITON-DiT。具体而言，VITON-DiT由一个服装提取器、一个时空去噪DiT和一个身份保持ControlNet组成。为了忠实地恢复服装细节，提取的服装特征与去噪DiT和ControlNet的自注意力输出相融合。我们还在训练期间引入了新颖的随机选择策略，并在推理时引入了插值自回归（IAR）技术，以促进长视频生成。与现有尝试需要费力且受限地构建配对训练数据集、严重限制其可扩展性不同，VITON-DiT通过仅依赖非配对的人类舞蹈视频和精心设计的多阶段训练策略来缓解这一问题。此外，我们整理了一个具有挑战性的基准数据集，以评估随意视频试穿的性能。大量实验证明了VITON-DiT在为具有复杂人体姿势的真实场景视频生成时空一致的试穿结果方面的优越性。
英文摘要：
Video try-on stands as a promising area for its tremendous real-world potential. Prior works are limited to transferring product clothing images onto person videos with simple poses and backgrounds, while underperforming on casually captured videos. Recently, Sora revealed the scalability of Diffusion Transformer (DiT) in generating lifelike videos featuring real-world scenarios. Inspired by this, we explore and propose the first DiT-based video try-on framework for practical in-the-wild applications, named VITON-DiT. Specifically, VITON-DiT consists of a garment extractor, a Spatial-Temporal denoising DiT, and an identity preservation ControlNet. To faithfully recover the clothing details, the extracted garment features are fused with the self-attention outputs of the denoising DiT and the ControlNet. We also introduce novel random selection strategies during training and an Interpolated Auto-Regressive (IAR) technique at inference to facilitate long video generation. Unlike existing attempts that require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability, VITON-DiT alleviates this by relying solely on unpaired human dance videos and a carefully designed multi-stage training strategy. Furthermore, we curate a challenging benchmark dataset to evaluate the performance of casual video try-on. Extensive experiments demonstrate the superiority of VITON-DiT in generating spatio-temporal consistent try-on results for in-the-wild videos with complicated human poses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
