We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Carnegie Mellon University（卡内基梅隆大学）；UiT The Arctic University of Norway（挪威北极大学）；ByteDance（字节跳动）
Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. In this work, we take a step forwards to explore versatile virtual try-on solutions, which we argue should possess three main properties, namely, they should support unsupervised training, arbitrary garment categories, and controllable garment editing. To this end, we propose a characteristic-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN++ (PASTA-GAN++), to achieve a versatile system for high-resolution unpaired virtual try-…
MuGa-VTON: Multi-Garment Virtual Try-On via Diffusion Transformers with Prompt Customization
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Khalifa University（哈利法大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Jio Institute（Jio研究所）
Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；UiT The Arctic University of Norway（挪威北极大学）；Peng Cheng Laboratory（鹏城实验室）
Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disentangling style and…
This paper presents a learning-based clothing animation method for highly efficient virtual try-on simulation. Given a garment, we preprocess a rich database of physically-based dressed character simulations, for multiple body shapes and animations. Then, using this database, we train a learning-based model of cloth drape and wrinkles, as a function of body shape and dynamics. We propose a model that separates global garment fit, due to body shape, from local garment wrinkles, due to both pose dynamics and body shape. We use a recurrent neural network to regress garment wrinkles, and we achiev…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We propose a flexible person generation framework called Dressing in Order (DiOr), which supports 2D pose transfer, virtual try-on, and several fashion editing tasks. The key to DiOr is a novel recurrent generation pipeline to sequentially put garments on a person, so that trying on the same garments in different orders will result in different looks. Our system can produce dressing effects not achievable by existing work, including different interactions of garments (e.g., wearing a top tucked into the bottom or over it), as well as layering of multiple garments of the same type (e.g., jacket…
In this paper, we develop a robust 3D garment digitization solution that can generalize well on real-world fashion catalog images with cloth texture occlusions and large body pose variations. We assumed fixed topology parametric template mesh models for known types of garments (e.g., T-shirts, Trousers) and perform mapping of high-quality texture from an input catalog image to UV map panels corresponding to the parametric mesh model of the garment. We achieve this by first predicting a sparse set of 2D landmarks on the boundary of the garments. Subsequently, we use these landmarks to perform T…
Virtual try-on is a promising application of computer graphics and human computer interaction that can have a profound real-world impact especially during this pandemic. Existing image-based works try to synthesize a try-on image from a single image of a target garment, but it inherently limits the ability to react to possible interactions. It is difficult to reproduce the change of wrinkles caused by pose and body size change, as well as pulling and stretching of the garment by hand. In this paper, we propose an alternative per garment capture and synthesis workflow to handle such rich intera…
Single Stage Warped Cloth Learning and Semantic-Contextual Attention Feature Fusion for Virtual TryOn
Bharti School of Telecommunications, Indian Institute of Technology, Delhi, India（印度德里印度理工学院巴蒂电信学院）；Indian Institute of Information Technology, Sonepat, Haryana, India（印度哈里亚纳邦索内帕特印度信息技术学院）；Department of Electrical Engineering, Indian Institute of Technology, Delhi, India（印度德里印度理工学院电气工程系）
Sun Yat-Sen University（中山大学）；ByteDance（字节跳动）；Carnegie Mellon University（卡内基梅隆大学）；Peng Cheng Laboratory（鹏城实验室）
Image-based Virtual Try-ON aims to transfer an in-shop garment onto a specific person. Existing methods employ a global warping module to model the anisotropic deformation for different garment parts, which fails to preserve the semantic information of different parts when receiving challenging inputs (e.g, intricate human poses, difficult garments). Moreover, most of them directly warp the input garment to align with the boundary of the preserved region, which usually requires texture squeezing to meet the boundary shape constraint and thus leads to texture distortion. The above inferior perf…
BooM-VVT: Boosting Mask-Free Video Virtual Try-On with Image-Level Pseudo Data
Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Meituan（美团）
Multi-pose virtual try-on (MPVTON) aims to fit a target garment onto a person at a target pose. Compared to traditional virtual try-on (VTON) that fits the garment but keeps the pose unchanged, MPVTON provides a better try-on experience, but is also more challenging due to the dual garment and pose editing objectives. Existing MPVTON methods adopt a pipeline comprising three disjoint modules including a target semantic layout prediction module, a coarse try-on image generator and a refinement try-on image generator. These models are trained separately, leading to sub-optimal model training and…
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Centre on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）
Image-based virtual try-on aims to fit an in-shop garment into a clothed person image. To achieve this, a key step is garment warping which spatially aligns the target garment with the corresponding body parts in the person image. Prior methods typically adopt a local appearance flow estimation model. They are thus intrinsically susceptible to difficult body poses/occlusions and large mis-alignments between person and garment images (see Fig.~\ref{fig:fig1}). To overcome this limitation, a novel global appearance flow estimation model is proposed in this work. For the first time, a StyleGAN ba…
Once Is Enough: Lightweight DiT-Based Video Virtual Try-On via One-Time Garment Appearance Injection
School of computer science, Shanghai key laboratory of data science, Fudan University, China（计算机学院、数据科学重点实验室、复旦大学）；Tencent Youtu Lab, China（腾讯优图实验室）；Shanghai Ocean University, China（上海海洋大学）
Neural Clothing Tryer: Customized Virtual Try-On via Semantic Enhancement and Controlling Diffusion Model
Guangdong University of Technology, Guangzhou 510006, China；Guangdong Provincial Key Laboratory of Intellectual Property \& Big Data, Guangzhou 510006, China；South China University of Technology, Guangzhou 510006, China；Genstoraige Technology (Beijing) Co., Ltd, Beijing 100000, China
Media Lab, Massachusetts Institute of Technology（麻省理工学院媒体实验室）；BITS Pilani（印度皮拉尼比尔拉技术与科学学院）；Media and Data Science Research Lab, Adobe（Adobe媒体与数据科学研究实验室）
Image-based virtual try-on involves synthesizing perceptually convincing images of a model wearing a particular garment and has garnered significant research interest due to its immense practical applicability. Recent methods involve a two stage process: i) warping of the garment to align with the model ii) texture fusion of the warped garment and target model to generate the try-on output. Issues arise due to the non-rigid nature of garments and the lack of geometric information about the model or the garment. It often results in improper rendering of granular details. We propose ZFlow, an en…
IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter
School of Software Engineering, South China University of Technology（华南理工大学软件学院）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Peng Cheng Laboratory（鹏城实验室）
TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models
Machine Learning Group, CITEC Bielefeld University Germany（比勒菲尔德大学机器学习小组、CITEC）；Institute of Mathematics Technical University Berlin Germany（柏林技术大学数学研究所）
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）；University of Washington（华盛顿大学）
Given a pair of images-target person and garment on another person-we automatically generate the target person in the given garment. Previous methods mostly focused on texture transfer via paired data training, while overlooking body shape deformations, skin color, and seamless blending of garment with the person. This work focuses on those three components, while also not requiring paired data training. We designed a pose conditioned StyleGAN2 architecture with a clothing segmentation branch that is trained on images of people wearing garments. Once trained, we propose a new layered latent sp…
