The last few years have witnessed the great success of non-linear generative models in synthesizing high-quality photorealistic face images. Many recent 3D facial texture reconstruction and pose manipulation from a single image approaches still rely on large and clean face datasets to train image-to-image Generative Adversarial Networks (GANs). Yet the collection of such a large scale high-resolution 3D texture dataset is still very costly and difficult to maintain age/ethnicity balance. Moreover, regression-based approaches suffer from generalization to the in-the-wild conditions and are unable to fine-tune to a target-image. In this work, we propose an unsupervised approach for one-shot 3D facial texture completion that does not require large-scale texture datasets, but rather harnesses the knowledge stored in 2D face generators. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator, based on the visible parts. Finally, we stitch the most visible textures at different angles in the UV image-plane. Further, we frontalize the target image by projecting the completed texture into the generator. The qualitative and quantitative experiments demonstrate that the completed UV textures and frontalized images are of high quality, resembles the original identity, can be used to train a texture GAN model for 3DMM fitting and improve pose-invariant face recognition.
Tsinghua University（清华大学）；University of Cincinnati（辛辛那提大学）
Visual surface inspection is a challenging task owing to the highly diverse appearance of target surfaces and defective regions. Previous attempts heavily rely on vast quantities of training examples with manual annotation. However, in some practical cases, it is difficult to obtain a large number of samples for inspection. To combat it, we propose a hierarchical texture-perceiving generative adversarial network (HTP-GAN) that is learned from the one-shot normal image in an unsupervised scheme. Specifically, the HTP-GAN contains a pyramid of convolutional GANs that can capture the global struc…
The Hong Kong University of Science and Technology（香港科技大学）；Huya Inc（虎牙公司）
We present a new method for few-shot human motion transfer that achieves realistic human image generation with only a small number of appearance inputs. Despite recent advances in single person motion transfer, prior methods often require a large number of training images and take long training time. One promising direction is to perform few-shot human motion transfer, which only needs a few of source images for appearance transfer. However, it is particularly challenging to obtain satisfactory transfer results. In this paper, we address this issue by rendering a human texture map to a surface…
Picsart AI Research (PAIR)（Picsart人工智能研究院）；Peking University（北京大学）
Image completion with large-scale free-form missing regions is one of the most challenging tasks for the computer vision community. While researchers pursue better solutions, drawbacks such as pattern unawareness, blurry textures, and structure distortion remain noticeable, and thus leave space for improvement. To overcome these challenges, we propose a new StyleGAN-based image completion network, Spectral Hint GAN (SH-GAN), inside which a carefully designed spectral processing module, Spectral Hint Unit, is introduced. We also propose two novel 2D spectral processing strategies, Heterogeneous…
Distance Weighted Trans Network for Image Completion
East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；University of Leicester（莱斯特大学）；Northwestern Polytechnical University（西北工业大学）
Consistent Zero-shot 3D Texture Synthesis Using Geometry-aware Diffusion and Temporal Video Models
Department of Image, Chung-Ang University（Chung-Ang大学图像系）；Graphics AI Lab, NCSOFT（NCSOFT图形AI实验室）；Department of Computer Science, University of Seoul（首尔大学计算机科学系）
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Pengcheng Laboratory, China（鹏城实验室）；Harbin Institute of Technology, China（哈尔滨工业大学）；Harbin Institute of Technology, Suzhou Research Institute, China（哈尔滨工业大学苏州市研究院）
Nanjing University of Science and Technology（南京理工大学）；Tencent AI Lab（腾讯人工智能实验室）
We present a large-scale facial UV-texture dataset that contains over 50,000 high-quality texture UV-maps with even illuminations, neutral expressions, and cleaned facial regions, which are desired characteristics for rendering realistic 3D face models under different lighting conditions. The dataset is derived from a large-scale face image dataset namely FFHQ, with the help of our fully automatic and robust UV-texture production pipeline. Our pipeline utilizes the recent advances in StyleGAN-based facial image editing approaches to generate multi-view normalized face images from single-image …
Advances in generative modeling based on GANs has motivated the community to find their use beyond image generation and editing tasks. In particular, several recent works have shown that GAN representations can be re-purposed for discriminative tasks such as part segmentation, especially when training data is limited. But how do these improvements stack-up against recent advances in self-supervised learning? Motivated by this we present an alternative approach based on contrastive learning and compare their performance on standard few-shot part segmentation benchmarks. Our experiments reveal t…
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Automatic generation of artistic glyph images is a challenging task that attracts many research interests. Previous methods either are specifically designed for shape synthesis or focus on texture transfer. In this paper, we propose a novel model, AGIS-Net, to transfer both shape and texture styles in one-stage with only a few stylized samples. To achieve this goal, we first disentangle the representations for content and style by using two encoders, ensuring the multi-content and multi-style generation. Then we utilize two collaboratively working decoders to generate the glyph shape image and…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Prior work to infer 3D texture use either texture atlases, which require uv-mappings and hence have discontinuities, or colored voxels, which are memory inefficient and limited in resolution. Recent work, predicts RGB color at every XYZ coordinate forming a texture field, but focus on completing texture given a single 2D image. Instead, we focus on 3D texture and geometry completion from partial and incomplete 3D scans. IF-Nets have recently achieved state-of-the-art results on 3D geometry completion using a multi-scale deep feature encoding, but the outputs lack texture. In this work, we gene…
ST Engineering - NTU Corporate Lab（ST工程-南洋理工大学企业实验室）；Nanyang Technological University（南洋理工大学）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；City University of Hong Kong（香港城市大学）
The light field faithfully records the spatial and angular configurations of the scene, which facilitates a wide range of imaging possibilities. In this work, we propose an LF synthesis algorithm which renders high quality novel LF views far outside the range of angular baselines of the given references. A stratified synthesis strategy is adopted which parses the scene content based on stratified disparity layers and across a varying range of spatial granularities. Such a stratified methodology proves to help preserve scene structures over large perspective shifts, and it provides informative …
NEC Laboratories America（美国NEC实验室）；University of Southern California（南加州大学）；University of California, San Diego（加州大学圣地亚哥分校）
We address the challenging problem of generating facial attributes using a single image in an unconstrained pose. In contrast to prior works that largely consider generation on 2D near-frontal images, we propose a GAN-based framework to generate attributes directly on a dense 3D representation given by UV texture and position maps, resulting in photorealistic, geometrically-consistent and identity-preserving outputs. Starting from a self-occluded UV texture map obtained by applying an off-the-shelf 3D reconstruction method, we propose two novel components. First, a texture completion generativ…
Reconstructing 3D human body shapes from 3D partial textured scans remains a fundamental task for many computer vision and graphics applications -- e.g., body animation, and virtual dressing. We propose a new neural network architecture for 3D body shape and high-resolution texture completion -- BCom-Net -- that can reconstruct the full geometry from mid-level to high-level partial input scans. We decompose the overall reconstruction task into two stages - first, a joint implicit learning network (SCom-Net and TCom-Net) that takes a voxelized scan and its occupancy grid as input to reconstruct…
One-shot Human Motion Transfer via Occlusion-Robust Flow Prediction and Neural Texturing
School of Computer Science and Technology, Guangdong University of Technology（广东工业大学计算机科学与技术学院）；University of Oxford（牛津大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Inpainting high-resolution images with large holes challenges existing deep learning based image inpainting methods. We present a novel framework -- PyramidFill for high-resolution image inpainting task, which explicitly disentangles content completion and texture synthesis. PyramidFill attempts to complete the content of unknown regions in a lower-resolution image, and synthesis the textures of unknown regions in a higher-resolution image, progressively. Thus, our model consists of a pyramid of fully convolutional GANs, wherein the content GAN is responsible for completing contents in the low…
Samsung AI Center（三星人工智能中心）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；University of Cambridge（剑桥大学）
We propose a neural rendering-based system that creates head avatars from a single photograph. Our approach models a person's appearance by decomposing it into two layers. The first layer is a pose-dependent coarse image that is synthesized by a small neural network. The second layer is defined by a pose-independent texture image that contains high-frequency details. The texture image is generated offline, warped and added to the coarse image to ensure a high effective resolution of synthesized head views. We compare our system to analogous state-of-the-art systems in terms of visual quality a…
We introduce one-shot texture segmentation: the task of segmenting an input image containing multiple textures given a patch of a reference texture. This task is designed to turn the problem of texture-based perceptual grouping into an objective benchmark. We show that it is straight-forward to generate large synthetic data sets for this task from a relatively small number of natural textures. In particular, this task can be cast as a self-supervised problem thereby alleviating the need for massive amounts of manually annotated data necessary for traditional segmentation tasks. In this paper w…
RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, China（网络与交换技术国家重点实验室，北京邮电大学，中国）；Midea Group, China（美的集团，中国）；School of Computer Science, Beijing University of Posts and Telecommunications, China（计算机科学学院，北京邮电大学，中国）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Snap Research（Snap研究院）
In this work, we present a novel framework built to simplify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, including images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multi-modal inputs, we employ task-specific encoders with dropout fol…
Cornell University, Cornell Tech（康奈尔大学康奈尔科技校区）；Google Research（谷歌研究院）；NVIDIA（英伟达）；University of Copenhagen（哥本哈根大学）
Recent advances in data augmentation enable one to translate images by learning the mapping between a source domain and a target domain. Existing methods tend to learn the distributions by training a model on a variety of datasets, with results evaluated largely in a subjective manner. Relatively few works in this area, however, study the potential use of image synthesis methods for recognition tasks. In this paper, we propose and explore the problem of image translation for data augmentation. We first propose a lightweight yet efficient model for translating texture to augment images based on…
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）；Manipal Institute of Technology Bengaluru（马尼帕尔理工学院（班加罗尔））
Forest plays a vital role in reducing greenhouse gas emissions and mitigating climate change besides maintaining the world's biodiversity. The existing satellite-based forest monitoring system utilizes supervised learning approaches that are limited to a particular region and depend on manually annotated data to identify forest. This work envisages forest identification as a few-shot semantic segmentation task to achieve generalization across different geographical regions. The proposed few-shot segmentation approach incorporates a texture attention module in the prototypical network to highli…
Southwest Jiaotong University（西南交通大学）；Capital Normal University（首都师范大学）
Photogrammetric mesh models obtained from aerial oblique images have been widely used for urban reconstruction. However, the photogrammetric meshes also suffer from severe texture problems, especially on the road areas due to occlusion. This paper proposes a structure-aware completion approach to improve the quality of meshes by removing undesired vehicles on the road seamlessly. Specifically, the discontinuous texture atlas is first integrated to a continuous screen space through rendering by the graphics pipeline; the rendering also records necessary mapping for deintegration to the original…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Megvii Technology（旷视科技）
Deep image completion usually fails to harmonically blend the restored image into existing content, especially in the boundary area. This paper handles with this problem from a new perspective of creating a smooth transition and proposes a concise Deep Fusion Network (DFNet). Firstly, a fusion block is introduced to generate a flexible alpha composition map for combining known and unknown regions. The fusion block not only provides a smooth fusion between restored and existing content, but also provides an attention map to make network focus more on the unknown pixels. In this way, it builds a…
South China University of Technology（华南理工大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Sydney（悉尼大学）
High-fidelity face completion is a challenging task due to the rich and subtle facial textures involved. What makes it more complicated is the correlations between different facial components, for example, the symmetry in texture and structure between both eyes. While recent works adopted the attention mechanism to learn the contextual relations among elements of the face, they have largely overlooked the disastrous impacts of inaccurate attention scores; in addition, they fail to pay sufficient attention to key facial components, the completion results of which largely determine the authentic…
City University of Hong Kong（香港城市大学）；Microsoft Cloud + AI（微软云与人工智能部门）
Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This…
Beijing University of Posts and Telecommunications（北京邮电大学）；Midea Group（美的集团）
The raw depth image captured by the indoor depth sensor usually has an extensive range of missing depth values due to inherent limitations such as the inability to perceive transparent objects and limited distance range. The incomplete depth map burdens many downstream vision tasks, and a rising number of depth completion methods have been proposed to alleviate this issue. While most existing methods can generate accurate dense depth maps from sparse and uniformly sampled depth maps, they are not suitable for complementing the large contiguous regions of missing depth values, which is common a…
Despite the initial belief that Convolutional Neural Networks (CNNs) are driven by shapes to perform visual recognition tasks, recent evidence suggests that texture bias in CNNs provides higher performing models when learning on large labeled training datasets. This contrasts with the perceptual bias in the human visual cortex, which has a stronger preference towards shape components. Perceptual differences may explain why CNNs achieve human-level performance when large labeled datasets are available, but their performance significantly degrades in lowlabeled data scenarios, such as few-shot s…
The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Ant Group（蚂蚁集团）；Institute of AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
Few-shot medical image classification with simple shape and texture text descriptors using vision-language models
RIKEN Center for Brain Science（理化学研究所脑科学中心）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Faculty of Computer Science, Universitas Indonesia（印度尼西亚大学计算机学院）
University of Science and Technology of China（中国科学技术大学）
In this paper, we tackle one-shot texture retrieval: given an example of a new reference texture, detect and segment all the pixels of the same texture category within an arbitrary image. To address this problem, we present an OS-TR network to encode both reference and query image, leading to achieve texture segmentation towards the reference category. Unlike the existing texture encoding methods that integrate CNN with orderless pooling, we propose a directionality-aware module to capture the texture variations at each direction, resulting in spatially invariant representation. To segment new…
