本文提出 MC-VTON，利用 DiT 骨干以仅人物与服装图像实现虚拟试衣，并通过高效微调和蒸馏扩散在8步内生成逼真结果。
摘要：
基于扩散模型的虚拟试衣方法能够实现逼真的试衣效果。它们使用额外的参考网络或附加图像编码器来处理多个条件图像输入，这增加了复杂的预处理和额外的计算成本。此外，它们需要超过25个推理步骤，导致更长的推理时间。在本工作中，随着扩散 Transformer（DiT）的发展，我们重新思考了附加参考网络或图像编码器的必要性，并提出 MC-VTON，它利用 DiT 的内在骨干网络无缝整合极简条件试衣输入。与现有方法相比，MC-VTON 的优越性体现在四个方面：（1）卓越的细节保真度。我们基于 DiT 的 MC-VTON 在保留细粒度细节方面表现出卓越的保真度。（2）简化的网络和输入。我们移除了任何额外的参考网络或图像编码器。我们还移除了不必要的条件，如长提示词、姿态估计、人体解析和深度图。我们只需要遮罩人物图像和服装图像。（3）参数高效训练。为处理试衣任务，我们微调 FLUX.1-dev，仅增加39.7M个额外参数（占骨干参数的0.33%）。（4）更少的推理步骤。我们在 MC-VTON 上应用蒸馏扩散，仅需8步即可生成逼真的试衣图像，且仅增加86.8M个额外参数（占骨干参数的0.72%）。实验表明，与基线方法相比，MC-VTON 以更少的条件输入、可训练参数和推理步骤取得了更优的定性和定量结果。
英文摘要：
Virtual try-on methods based on diffusion models achieve realistic try-on effects. They use an extra reference network or an additional image encoder to process multiple conditional image inputs, which adds complexity pre-processing and additional computational costs. Besides, they require more than 25 inference steps, bringing longer inference time. In this work, with the development of diffusion transformer (DiT), we rethink the necessity of additional reference network or image encoder and introduce MC-VTON, which leverages DiT's intrinsic backbone to seamlessly integrate minimal conditional try-on inputs. Compared to existing methods, the superiority of MC-VTON is demonstrated in four aspects: (1) Superior detail fidelity. Our DiT-based MC-VTON exhibits superior fidelity in preserving fine-grained details. (2) Simplified network and inputs. We remove any extra reference network or image encoder. We also remove unnecessary conditions like the long prompt, pose estimation, human parsing, and depth map. We require only the masked person image and the garment image. (3) Parameter-efficient training. To process the try-on task, we fine-tune the FLUX.1-dev with only 39.7M additional parameters (0.33% of the backbone parameters). (4) Less inference steps. We apply distillation diffusion on MC-VTON and only need 8 steps to generate a realistic try-on image, with only 86.8M additional parameters (0.72% of the backbone parameters). Experiments show that MC-VTON achieves superior qualitative and quantitative results with fewer condition inputs, trainable parameters, and inference steps than baseline methods.
FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Suzhou Xiangji Technology Service Co., Ltd.（苏州湘记科技服务有限公司）
Sun Yat-sen University（中山大学）；South China Normal University（华南师范大学）
Virtual try-on system under arbitrary human poses has huge application potential, yet raises quite a lot of challenges, e.g. self-occlusions, heavy misalignment among diverse poses, and diverse clothes textures. Existing methods aim at fitting new clothes into a person can only transfer clothes on the fixed human pose, but still show unsatisfactory performances which often fail to preserve the identity, lose the texture details, and decrease the diversity of poses. In this paper, we make the first attempt towards multi-pose guided virtual try-on system, which enables transfer clothes on a pers…
Multi-pose virtual try-on (MPVTON) aims to fit a target garment onto a person at a target pose. Compared to traditional virtual try-on (VTON) that fits the garment but keeps the pose unchanged, MPVTON provides a better try-on experience, but is also more challenging due to the dual garment and pose editing objectives. Existing MPVTON methods adopt a pipeline comprising three disjoint modules including a target semantic layout prediction module, a coarse try-on image generator and a refinement try-on image generator. These models are trained separately, leading to sub-optimal model training and…
TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models
Machine Learning Group, CITEC Bielefeld University Germany（比勒菲尔德大学机器学习小组、CITEC）；Institute of Mathematics Technical University Berlin Germany（柏林技术大学数学研究所）
Neural Clothing Tryer: Customized Virtual Try-On via Semantic Enhancement and Controlling Diffusion Model
Guangdong University of Technology, Guangzhou 510006, China；Guangdong Provincial Key Laboratory of Intellectual Property \& Big Data, Guangzhou 510006, China；South China University of Technology, Guangzhou 510006, China；Genstoraige Technology (Beijing) Co., Ltd, Beijing 100000, China
DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；ByteDance（字节跳动）；Carnegie Mellon University（卡内基梅隆大学）；Research Institute of Multiple Agents and Embodied Intelligence（多智能体与具身智能研究院）；Peng Cheng Laboratory（鹏城实验室）
本文提出 DreamVTON 模型，通过结合个性化扩散模型、Densepose-guided ControlNet 和基于模板的优化机制，分别优化 3D 人体的几何与纹理，解决了个性化模块削弱多视图合成能力的问题，实现了高质量的定制化 3D 虚拟试穿。
The virtual try-on system has gained great attention due to its potential to give customers a realistic, personalized product presentation in virtualized settings. In this paper, we present PT-VTON, a novel pose-transfer-based framework for cloth transfer that enables virtual try-on with arbitrary poses. PT-VTON can be applied to the fashion industry within minimal modification of existing systems while satisfying the overall visual fashionability and detailed fabric appearance requirements. It enables efficient clothes transferring between model and user images with arbitrary pose and body sh…
Image-based virtual try-on techniques have shown great promise for enhancing the user-experience and improving customer satisfaction on fashion-oriented e-commerce platforms. However, existing techniques are currently still limited in the quality of the try-on results they are able to produce from input images of diverse characteristics. In this work, we propose a Context-Driven Virtual Try-On Network (C-VTON) that addresses these limitations and convincingly transfers selected clothing items to the target subjects even under challenging pose configurations and in the presence of self-occlusio…
Image-based virtual try-on is challenging in fitting a target in-shop clothes into a reference person under diverse human poses. Previous works focus on preserving clothing details ( e.g., texture, logos, patterns ) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which could fit the desired clothing into a reference person und…
Shenzhen Campus of SYSU（中山大学深圳校区）；UiT The Arctic University of Norway（挪威北极大学）；UC San Diego（加利福尼亚大学圣迭戈分校）；Momo（陌陌科技）
Virtual 3D try-on can provide an intuitive and realistic view for online shopping and has a huge potential commercial value. However, existing 3D virtual try-on methods mainly rely on annotated 3D human shapes and garment templates, which hinders their applications in practical scenarios. 2D virtual try-on approaches provide a faster alternative to manipulate clothed humans, but lack the rich and realistic 3D representation. In this paper, we propose a novel Monocular-to-3D Virtual Try-On Network (M3D-VTON) that builds on the merits of both 2D and 3D approaches. By integrating 2D information e…
Sun Yat-Sen University（中山大学）；UiT The Arctic University of Norway（挪威北极大学）；Momo Technology Company Limited（陌陌科技有限公司）；DarkMatter AI Research（暗物质人工智能研究院）
Despite recent progress on image-based virtual try-on, current methods are constraint by shared warping networks and thus fail to synthesize natural try-on results when faced with clothing categories that require different warping operations. In this paper, we address this problem by finding clothing category-specific warping networks for the virtual try-on task via Neural Architecture Search (NAS). We introduce a NAS-Warping Module and elaborately design a bilevel hierarchical search space to identify the optimal network-level and operation-level flow estimation architecture. Given the networ…
South China University of Technology（华南理工大学）；Nanyang Technological University（南洋理工大学）；Guangdong University of Technology（广东工业大学）
With the development of Generative Adversarial Network, image-based virtual try-on methods have made great progress. However, limited work has explored the task of video-based virtual try-on while it is important in real-world applications. Most existing video-based virtual try-on methods usually require clothing templates and they can only generate blurred and low-resolution results. To address these challenges, we propose a Memory-based Video virtual Try-On Network (MV-TON), which seamlessly transfers desired clothes to a target person without using any clothing templates and generates high-…
The University of Tokyo（东京大学）；The International Research Center for Neurointelligence（国际神经智能研究中心）
Generating a virtual try-on image from in-shop clothing images and a model person's snapshot is a challenging task because the human body and clothes have high flexibility in their shapes. In this paper, we develop a Virtual Try-on Generative Adversarial Network (VITON-GAN), that generates virtual try-on images using images of in-shop clothing and a model person. This method enhances the quality of the generated image when occlusion is present in a model person's image (e.g., arms crossed in front of the clothes) by adding an adversarial mechanism in the training pipeline.
MOFA-VTON: More Fashion Possibilities with Fine-Grained Adaptations in Virtual Try-On
Harbin Institute of Technology（哈尔滨工业大学）；HiDream.ai Inc.（HiDream.ai公司）；Harbin Institute of Technology (Weihai) Qingdao Research Institute（哈尔滨工业大学（威海）青岛研究院）
Previous virtual try-on methods usually focus on aligning a clothing item with a person, limiting their ability to exploit the complex pose, shape and skin color of the person, as well as the overall structure of the clothing, which is vital to photo-realistic virtual try-on. To address this potential weakness, we propose a fill in fabrics (FIFA) model, a self-supervised conditional generative adversarial network based framework comprised of a Fabricator and a unified virtual try-on pipeline with a Segmenter, Warper and Fuser. The Fabricator aims to reconstruct the clothing image when provided…
MuGa-VTON: Multi-Garment Virtual Try-On via Diffusion Transformers with Prompt Customization
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Khalifa University（哈利法大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Jio Institute（Jio研究所）
