A deep learning based interactive sketching system for fashion images design
DexForce Technology Co. Ltd.（德克斯福科技有限公司）；Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；SmartMore Corporation（思谋科技有限公司）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Peng Cheng Lab（鹏程实验室）
Yao Li, Xianggang Yu, Xiaoguang Han, Nianjuan Jiang, Kui Jia, Jiangbo Lu
In this work, we propose an interactive system to design diverse high-quality garment images from fashion sketches and the texture information. The major challenge behind this system is to generate high-quality and detailed texture according to the user-provided texture information. Prior works mainly use the texture patch representation and try to map a small texture patch to a whole garment image, hence unable to generate high-quality details. In contrast, inspired by intrinsic image decomposition, we decompose this task into texture synthesis and shading enhancement. In particular, we propose a novel bi-colored edge texture representation to synthesize textured garment images and a shading enhancer to render shading based on the grayscale edges. The bi-colored edge representation provides simple but effective texture cues and color constraints, so that the details can be better reconstructed. Moreover, with the rendered shading, the synthesized garment image becomes more vivid.
From Air to Wear: Personalized 3D Digital Fashion with AR/VR Immersive 3D Sketching
Huzhou University（湖州大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Science and Technology of China（中国科学技术大学）；Moxin (Huzhou) Technology Co., LTD.（魔芯（湖州）科技有限公司）；Zhejiang University（浙江大学）
本文提出由 AR/VR 中 3D 草图驱动的服装生成框架，结合条件扩散模型、共享潜空间草图编码器与课程学习，并构建 KO3DClothes 数据集，显著提升数字服装设计的保真度和易用性。
Keyword-based mobile screen search does not account for screen content and fails to operate as a universal tool for all levels of users. Visual searching (e.g., image, sketch) is structured and easy to adopt. Current visual search approaches count on a complete screen and are therefore slow and tedious. PSDoodle employs a deep neural network to recognize partial screen element drawings instantly on a digital drawing interface and shows results in real-time. PSDoodle is the first tool that utilizes partial sketches and searches for screens in an interactive iterative way. PSDoodle supports diff…
It can be easy and even fun to sketch humans in different poses. In contrast, creating those same poses on a 3D graphics "mannequin" is comparatively tedious. Yet 3D body poses are necessary for various downstream applications. We seek to preserve the convenience of 2D sketching while giving users of different skill levels the flexibility to accurately and more quickly pose\slash refine a 3D mannequin. At the core of the interactive system, we propose a machine-learning model for inferring the 3D pose of a CG mannequin from sketches of humans drawn in a cylinder-person style. Training such a m…
University of the Philippines, Diliman（菲律宾大学迪利曼分校）
The rise of deep learning applications in the fashion industry has fueled advances in curating large-scale datasets to build applications for product design, image retrieval, and recommender systems. In this paper, the author proposes using Variational Autoencoder (VAE) to build an interactive fashion product application framework that allows the users to generate products with attributes according to their liking, retrieve similar styles for the same product category, and receive content-based recommendations from other categories. Fashion product images dataset containing eyewear, footwear, …
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；UT Austin（德克萨斯大学奥斯汀分校）；Pryon
Conversational interfaces for the detail-oriented retail fashion domain are more natural, expressive, and user friendly than classical keyword-based search interfaces. In this paper, we introduce the Fashion IQ dataset to support and advance research on interactive fashion image retrieval. Fashion IQ is the first fashion dataset to provide human-generated captions that distinguish similar pairs of garment images together with side-information consisting of real-world product descriptions and derived visual attribute labels for these images. We provide a detailed analysis of the characteristics…
Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel C…
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；The University of Hong Kong（香港大学）
In this paper, we propose the first sketching system for interactively personalized and photorealistic face caricaturing. Input an image of a human face, the users can create caricature photos by manipulating its facial feature curves. Our system firstly performs exaggeration on the recovered 3D face model according to the edited sketches, which is conducted by assigning the laplacian of each vertex a scaling factor. To construct the mapping between 2D sketches and a vertex-wise scaling field, a novel deep learning architecture is developed. With the obtained 3D caricature model, two images ar…
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；City University of Hong Kong（香港城市大学）；University of Science and Technology of China（中国科学技术大学）
Head shapes play an important role in 3D character design. In this work, we propose SimpModeling, a novel sketch-based system for helping users, especially amateur users, easily model 3D animalmorphic heads - a prevalent kind of heads in character design. Although sketching provides an easy way to depict desired shapes, it is challenging to infer dense geometric information from sparse line drawings. Recently, deepnet-based approaches have been taken to address this challenge and try to produce rich geometric details from very few strokes. However, while such methods reduce users' workload, th…
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学帕克分校）
We present a novel system for sketch-based face image editing, enabling users to edit images intuitively by sketching a few strokes on a region of interest. Our interface features tools to express a desired image manipulation by providing both geometry and color constraints as user-drawn strokes. As an alternative to the direct user input, our proposed system naturally supports a copy-paste mode, which allows users to edit a given image region by using parts of another exemplar image without the need of hand-drawn sketching at all. The proposed interface runs in real-time and facilitates an in…
Being able to forecast the popularity of new garment designs is very important in an industry as fast paced as fashion, both in terms of profitability and reducing the problem of unsold inventory. Here, we attempt to address this task in order to provide informative forecasts to fashion designers within a virtual reality designer application that will allow them to fine tune their creations based on current consumer preferences within an interactive and immersive environment. To achieve this we have to deal with the following central challenges: (1) the proposed method should not hinder the cr…
Interactive Garment Recommendation with User in the Loop
University of Siena（锡耶纳大学）；School of Software, Shandong University（山东大学软件学院）；University of Florence（佛罗伦萨大学）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）
We propose a new approach to interactive full-image semantic segmentation which enables quickly collecting training data for new datasets with previously unseen semantic classes (A demo is available at https://youtu.be/yUk8D5gEX-o). We leverage a key observation: propagation from labeled to unlabeled pixels does not necessarily require class-specific knowledge, but can be done purely based on appearance similarity within an image. We build on this observation and propose an approach capable of jointly propagating pixel labels from multiple classes without having explicit class-specific appeara…
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Wuhan University of Technology（武汉理工大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this demo, we present Chat-to-Design, a new multimodal interaction system for personalized fashion design. Compared to classic systems that recommend apparel based on keywords, Chat-to-Design enables users to design clothes in two steps: 1) coarse-grained selection via conversation and 2) fine-grained editing via an interactive interface. It encompasses three sub-systems to deliver an immersive user experience: A conversation system empowered by natural language understanding to accept users' requests and manages dialogs; A multimodal fashion retrieval system empowered by a large-scale pret…
Sun Yat-sen University（中山大学）；Petuum Inc（Petuum公司）；ByteDance AI Lab（字节跳动AI实验室）
Interactive fashion image manipulation, which enables users to edit images with sketches and color strokes, is an interesting research problem with great application value. Existing works often treat it as a general inpainting task and do not fully leverage the semantic structural information in fashion images. Moreover, they directly utilize conventional convolution and normalization layers to restore the incomplete image, which tends to wash away the sketch and color information. In this paper, we propose a novel Fashion Editing Generative Adversarial Network (FE-GAN), which is capable of ma…
Oslo University Hospital（奥斯陆大学医院）；Mayo Clinic（梅奥诊所）；Nvidia Corporation（英伟达公司）
Image segmentation plays an essential role in medicine for both diagnostic and interventional tasks. Segmentation approaches are either manual, semi-automated or fully-automated. Manual segmentation offers full control over the quality of the results, but is tedious, time consuming and prone to operator bias. Fully automated methods require no human effort, but often deliver sub-optimal results without providing users with the means to make corrections. Semi-automated approaches keep users in control of the results by providing means for interaction, but the main challenge is to offer a good t…
University College London（伦敦大学学院）；Royal College of Art（皇家艺术学院）
The rise of Machine Learning (ML) is gradually digitalizing and reshaping the fashion industry. Recent years have witnessed a number of fashion AI applications, for example, virtual try-ons. Textile material identification and categorization play a crucial role in the fashion textile sector, including fashion design, retails, and recycling. At the same time, Net Zero is a global goal and the fashion industry is undergoing a significant change so that textile materials can be reused, repaired and recycled in a sustainable manner. There is still a challenge in identifying textile materials autom…
In this paper, we present the texture reformer, a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. The challenges lie in three aspects: 1) the diversity of tasks, 2) the simplicity of guidance maps, and 3) the execution efficiency. To address these challenges, our key idea is to use a novel feed-forward multi-view and multi-stage synthesis procedure consisting of I) a global view structure alignment stage, II) a local view texture refinement stage, and III) a holistic effect enhancement stage to synthesize high-quality results with cohere…
DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing
Shenzhen Technology University（深圳科技大学）；University of Washington（华盛顿大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）
Text-based image retrieval has seen considerable progress in recent years. However, the performance of existing methods suffers in real life since the user is likely to provide an incomplete description of an image, which often leads to results filled with false positives that fit the incomplete description. In this work, we introduce the partial-query problem and extensively analyze its influence on text-based image retrieval. Previous interactive methods tackle the problem by passively receiving users' feedback to supplement the incomplete query iteratively, which is time-consuming and requi…
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所IIS）；Nuremberg Institute of Technology Georg Simon Ohm（纽伦堡乔治·西蒙·欧姆技术学院）
Would you trust physicians if they cannot explain their decisions to you? Medical diagnostics using machine learning gained enormously in importance within the last decade. However, without further enhancements many state-of-the-art machine learning methods are not suitable for medical application. The most important reasons are insufficient data set quality and the black-box behavior of machine learning algorithms such as Deep Learning models. Consequently, end-users cannot correct the model's decisions and the corresponding explanations. The latter is crucial for the trustworthiness of machi…
Microsoft Dynamics 365 AI Research（微软 Dynamics 365 人工智能研究院）；Duke University（杜克大学）；Microsoft Research（微软研究院）
Most existing text-to-image synthesis tasks are static single-turn generation, based on pre-defined textual descriptions of images. To explore more practical and interactive real-life applications, we introduce a new task - Interactive Image Editing, where users can guide an agent to edit images via multi-turn textual commands on-the-fly. In each session, the agent takes a natural language description from the user as the input and modifies the image generated in the previous turn to a new design, following the user description. The main challenges in this sequential and interactive image gene…
Interactive Visual Assessment for Text-to-Image Generation Models
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；National Cheng Kung University（成功大学）
Tel Aviv University（特拉维夫大学）；Cornell Tech, Cornell University（康奈尔科技学院，康奈尔大学）
As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over the texture appearance. To encourage fidelity to the input condition, we introduce a novel differen…
ChatGarment: Garment Estimation, Generation and Editing via Large Language Models
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Shanghai Jiao Tong University（上海交通大学）；EPFL（洛桑联邦理工学院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）；Mila, University of Montreal（蒙特利尔大学米拉研究所）；Meshcapade（Meshcapade公司）；Stanford University（斯坦福大学）
University of Illinois Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Revery AI Inc.（Revery AI公司）；JD AI Research（京东人工智能研究院）
Virtual try-on methods aim to generate images of fashion models wearing arbitrary combinations of garments. This is a challenging task because the generated image must appear realistic and accurately display the interaction between garments. Prior works produce images that are filled with artifacts and fail to capture important visual details necessary for commercial applications. We propose Outfit Visualization Net (OVNet) to capture these important details (e.g. buttons, shading, textures, realistic hemlines, and interactions between garments) and produce high quality multiple-garment virtua…
