We present a new method for few-shot human motion transfer that achieves realistic human image generation with only a small number of appearance inputs. Despite recent advances in single person motion transfer, prior methods often require a large number of training images and take long training time. One promising direction is to perform few-shot human motion transfer, which only needs a few of source images for appearance transfer. However, it is particularly challenging to obtain satisfactory transfer results. In this paper, we address this issue by rendering a human texture map to a surface geometry (represented as a UV map), which is personalized to the source person. Our geometry generator combines the shape information from source images, and the pose information from 2D keypoints to synthesize the personalized UV map. A texture generator then generates the texture map conditioned on the texture of source images to fill out invisible parts. Furthermore, we may fine-tune the texture map on the manifold of the texture generator from a few source images at the test time, which improves the quality of the texture map without over-fitting or artifacts. Extensive experiments show the proposed method outperforms state-of-the-art methods both qualitatively and quantitatively. Our code is available at https://github.com/HuangZhiChao95/FewShotMotionTransfer.
Modern 3D-GANs synthesize geometry and texture by training on large-scale datasets with a consistent structure. Training such models on stylized, artistic data, with often unknown, highly variable geometry, and camera information has not yet been shown possible. Can we train a 3D GAN on such artistic data, while maintaining multi-view consistency and texture quality? To this end, we propose an adaptation framework, where the source domain is a pre-trained 3D-GAN, while the target domain is a 2D-GAN trained on artistic datasets. We then distill the knowledge from a 2D generator to the source 3D…
UVMap-ID: A Controllable and Personalized UV Map Generative Model
University of Trento（特伦托大学）；Ocean University of China（中国海洋大学）；ETH Zürich（苏黎世联邦理工学院）；PicsArt AI Research（PicsArt人工智能研究院）；Georgia Institute of Technology（佐治亚理工学院）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
本文提出 UVMap-ID，通过微调集成人脸融合模块的文本到图像扩散模型，并利用带文本与 Face ID 标注的小规模属性均衡数据集，实现可控、个性化的人物 UV Map 生成与多维质量评估。
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Pengcheng Laboratory, China（鹏城实验室）；Harbin Institute of Technology, China（哈尔滨工业大学）；Harbin Institute of Technology, Suzhou Research Institute, China（哈尔滨工业大学苏州市研究院）
Consistent Zero-shot 3D Texture Synthesis Using Geometry-aware Diffusion and Temporal Video Models
Department of Image, Chung-Ang University（Chung-Ang大学图像系）；Graphics AI Lab, NCSOFT（NCSOFT图形AI实验室）；Department of Computer Science, University of Seoul（首尔大学计算机科学系）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Automatic generation of artistic glyph images is a challenging task that attracts many research interests. Previous methods either are specifically designed for shape synthesis or focus on texture transfer. In this paper, we propose a novel model, AGIS-Net, to transfer both shape and texture styles in one-stage with only a few stylized samples. To achieve this goal, we first disentangle the representations for content and style by using two encoders, ensuring the multi-content and multi-style generation. Then we utilize two collaboratively working decoders to generate the glyph shape image and…
The last few years have witnessed the great success of non-linear generative models in synthesizing high-quality photorealistic face images. Many recent 3D facial texture reconstruction and pose manipulation from a single image approaches still rely on large and clean face datasets to train image-to-image Generative Adversarial Networks (GANs). Yet the collection of such a large scale high-resolution 3D texture dataset is still very costly and difficult to maintain age/ethnicity balance. Moreover, regression-based approaches suffer from generalization to the in-the-wild conditions and are unab…
Tsinghua University（清华大学）；University of Cincinnati（辛辛那提大学）
Visual surface inspection is a challenging task owing to the highly diverse appearance of target surfaces and defective regions. Previous attempts heavily rely on vast quantities of training examples with manual annotation. However, in some practical cases, it is difficult to obtain a large number of samples for inspection. To combat it, we propose a hierarchical texture-perceiving generative adversarial network (HTP-GAN) that is learned from the one-shot normal image in an unsupervised scheme. Specifically, the HTP-GAN contains a pyramid of convolutional GANs that can capture the global struc…
Sun Yat-Sen University（中山大学）；National University of Singapore（新加坡国立大学）；QiHoo 360 Artificial intelligence institute（奇虎360人工智能研究院）
Face age progression, which aims to predict the future looks, is important for various applications and has been received considerable attentions. Existing methods and datasets are limited in exploring the effects of occupations which may influence the personal appearances. In this paper, we firstly introduce an occupational face aging dataset for studying the influences of occupations on the appearances. It includes five occupations, which enables the development of new algorithms for age progression and facilitate future researches. Second, we propose a new occupational-aware adversarial fac…
DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；ByteDance（字节跳动）；Carnegie Mellon University（卡内基梅隆大学）；Research Institute of Multiple Agents and Embodied Intelligence（多智能体与具身智能研究院）；Peng Cheng Laboratory（鹏城实验室）
本文提出 DreamVTON 模型，通过结合个性化扩散模型、Densepose-guided ControlNet 和基于模板的优化机制，分别优化 3D 人体的几何与纹理，解决了个性化模块削弱多视图合成能力的问题，实现了高质量的定制化 3D 虚拟试穿。
Télécom-Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎理工学院）；Ecole Polytechnique（巴黎综合理工学院）；CNRS（法国国家科学研究中心）
Adapting a segmentation model from a labeled source domain to a target domain, where a single unlabeled datum is available, is one the most challenging problems in domain adaptation and is otherwise known as one-shot unsupervised domain adaptation (OSUDA). Most of the prior works have addressed the problem by relying on style transfer techniques, where the source images are stylized to have the appearance of the target domain. Departing from the common notion of transferring only the target ``texture'' information, we leverage text-to-image diffusion models (e.g., Stable Diffusion) to generate…
DreamShot: Personalized Storyboard Synthesis with Video Diffusion Prior
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；ByteDance Intelligent Creation（字节跳动智能创作）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）
Samsung AI Center（三星人工智能中心）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；University of Cambridge（剑桥大学）
We propose a neural rendering-based system that creates head avatars from a single photograph. Our approach models a person's appearance by decomposing it into two layers. The first layer is a pose-dependent coarse image that is synthesized by a small neural network. The second layer is defined by a pose-independent texture image that contains high-frequency details. The texture image is generated offline, warped and added to the coarse image to ensure a high effective resolution of synthesized head views. We compare our system to analogous state-of-the-art systems in terms of visual quality a…
University of Southern California（南加州大学）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Pinscreen
We present a deep learning-based framework for portrait reenactment from a single picture of a target (one-shot) and a video of a driving subject. Existing facial reenactment methods suffer from identity mismatch and produce inconsistent identities when a target and a driving subject are different (cross-subject), especially in one-shot settings. In this work, we aim to address identity preservation in cross-subject portrait reenactment from a single picture. We introduce a novel technique that can disentangle identity from expressions and poses, allowing identity preserving portrait reenactme…
Advances in generative modeling based on GANs has motivated the community to find their use beyond image generation and editing tasks. In particular, several recent works have shown that GAN representations can be re-purposed for discriminative tasks such as part segmentation, especially when training data is limited. But how do these improvements stack-up against recent advances in self-supervised learning? Motivated by this we present an alternative approach based on contrastive learning and compare their performance on standard few-shot part segmentation benchmarks. Our experiments reveal t…
CusEnhancer: A Zero-Shot Scene and Controllability Enhancement Method for Photo Customization via ResInversion
College of Computer Science and Technology, East China University of Science and Technology（计算机科学与技术学院，东华大学）；Carnegie Mellon University（卡内基梅隆大学）；Microsoft（微软公司）
FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models
Ludwig Maximilian University of Munich（慕尼黑大学）；Siemens Technology（西门子技术公司）；University of Oxford（牛津大学）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Cornell University, Cornell Tech（康奈尔大学康奈尔科技校区）；Google Research（谷歌研究院）；NVIDIA（英伟达）；University of Copenhagen（哥本哈根大学）
Recent advances in data augmentation enable one to translate images by learning the mapping between a source domain and a target domain. Existing methods tend to learn the distributions by training a model on a variety of datasets, with results evaluated largely in a subjective manner. Relatively few works in this area, however, study the potential use of image synthesis methods for recognition tasks. In this paper, we propose and explore the problem of image translation for data augmentation. We first propose a lightweight yet efficient model for translating texture to augment images based on…
We introduce one-shot texture segmentation: the task of segmenting an input image containing multiple textures given a patch of a reference texture. This task is designed to turn the problem of texture-based perceptual grouping into an objective benchmark. We show that it is straight-forward to generate large synthetic data sets for this task from a relatively small number of natural textures. In particular, this task can be cast as a self-supervised problem thereby alleviating the need for massive amounts of manually annotated data necessary for traditional segmentation tasks. In this paper w…
Accurate image classification given small amounts of labelled data (few-shot classification) remains an open problem in computer vision. In this work we examine how the known texture bias of Convolutional Neural Networks (CNNs) affects few-shot classification performance. Although texture bias can help in standard image classification, in this work we show it significantly harms few-shot classification performance. After correcting this bias we demonstrate state-of-the-art performance on the competitive miniImageNet task using a method far simpler than the current best performing few-shot lear…
University of Science and Technology of China（中国科学技术大学）
In this paper, we tackle one-shot texture retrieval: given an example of a new reference texture, detect and segment all the pixels of the same texture category within an arbitrary image. To address this problem, we present an OS-TR network to encode both reference and query image, leading to achieve texture segmentation towards the reference category. Unlike the existing texture encoding methods that integrate CNN with orderless pooling, we propose a directionality-aware module to capture the texture variations at each direction, resulting in spatially invariant representation. To segment new…
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）；Manipal Institute of Technology Bengaluru（马尼帕尔理工学院（班加罗尔））
Forest plays a vital role in reducing greenhouse gas emissions and mitigating climate change besides maintaining the world's biodiversity. The existing satellite-based forest monitoring system utilizes supervised learning approaches that are limited to a particular region and depend on manually annotated data to identify forest. This work envisages forest identification as a few-shot semantic segmentation task to achieve generalization across different geographical regions. The proposed few-shot segmentation approach incorporates a texture attention module in the prototypical network to highli…
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；The University of Hong Kong（香港大学）
In this paper, we propose the first sketching system for interactively personalized and photorealistic face caricaturing. Input an image of a human face, the users can create caricature photos by manipulating its facial feature curves. Our system firstly performs exaggeration on the recovered 3D face model according to the edited sketches, which is conducted by assigning the laplacian of each vertex a scaling factor. To construct the mapping between 2D sketches and a vertex-wise scaling field, a novel deep learning architecture is developed. With the obtained 3D caricature model, two images ar…
We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category provisioned with a single sketch with a given part annotation while (i) preserving the parts semantics embedded in the exemplar, and (ii) being robust to input style and abstraction. We refer to this scenario as personalized. With that, we importantly enable a much-desired personalization capability for downstream fine-grained sketch analysis tasks. To train a robust segmentation module, we deform the exemplar sketch to each of the available sketches of the same …
Enhancing Zero-shot Personalized Image Aesthetics Assessment with Profile-aware Multimodal LLM
Mashang Consumer Finance Co., Ltd.（马莎消费金融有限公司）；Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
CVIT, IIIT Hyderabad（海得拉巴国际信息技术研究所视觉信息技术中心）；University of Bath（巴斯大学）
Lipreading or visually recognizing speech from the mouth movements of a speaker is a challenging and mentally taxing task. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from Amyotrophic Lateral Sclerosis (ALS) often lose muscle control, consequently their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting a large-scale dataset of a patient, needed to…
Samsung AI Center – Moscow（三星人工智能中心（莫斯科））；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Yandex Armenia（亚美尼亚扬德克斯公司）
We present a system for realistic one-shot mesh-based human head avatars creation, ROME for short. Using a single photograph, our model estimates a person-specific head mesh and the associated neural texture, which encodes both local photometric and geometric details. The resulting avatars are rigged and can be rendered using a neural network, which is trained alongside the mesh and texture estimators on a dataset of in-the-wild videos. In the experiments, we observe that our system performs competitively both in terms of head geometry recovery and the quality of renders, especially for the cr…
