In this paper, we propose an efficient method for robust 3D self-portraits using a single RGBD camera. Benefiting from the proposed PIFusion and lightweight bundle adjustment algorithm, our method can generate detailed 3D self-portraits in seconds and shows the ability to handle subjects wearing extremely loose clothes. To achieve highly efficient and robust reconstruction, we propose PIFusion, which combines learning-based 3D recovery with volumetric non-rigid fusion to generate accurate sparse partial scans of the subject. Moreover, a non-rigid volumetric deformation method is proposed to continuously refine the learned shape prior. Finally, a lightweight bundle adjustment algorithm is proposed to guarantee that all the partial scans can not only "loop" with each other but also remain consistent with the selected live key observations. The results and experiments show that the proposed method achieves more robust and efficient 3D self-portraits compared with state-of-the-art methods.
One of the most famous drawings by Leonardo da Vinci is a self-portrait in red chalk, where he looks quite old. In fact, there is a sketch in one of his notebooks, partially covered by written notes, that can be a self-portrait of the artist when he was young. The use of image processing, to remove the handwritten text and improve the image, allows a comparison of the two portraits.
CPqD Foundation（CPqD基金会）；University of Campinas (Unicamp)（坎皮纳斯州立大学）；Phelcom Technologies
Cross-domain biometrics has been emerging as a new necessity, which poses several additional challenges, including harsh illumination changes, noise, pose variation, among others. In this paper, we explore approaches to cross-domain face verification, comparing self-portrait photographs ("selfies") to ID documents. We approach the problem with proper image photometric adjustment and data standardization techniques, along with deep learning methods to extract the most prominent features from the data, reducing the effects of domain shift in this problem. We validate the methods using a novel da…
KU Leuven（鲁汶大学）；Adobe Research（Adobe研究院）；UC Berkeley（加州大学伯克利分校）
Due to the ubiquity of smartphones, it is popular to take photos of one's self, or "selfies." Such photos are convenient to take, because they do not require specialized equipment or a third-party photographer. However, in selfies, constraints such as human arm length often make the body pose look unnatural. To address this issue, we introduce $\textit{unselfie}$, a novel photographic transformation that automatically translates a selfie into a neutral-pose portrait. To achieve this, we first collect an unpaired dataset, and introduce a way to synthesize paired training data for self-supervise…
We resolve the ill-posed alpha matting problem from a completely different perspective. Given an input portrait image, instead of estimating the corresponding alpha matte, we focus on the other end, to subtly enhance this input so that the alpha matte can be easily estimated by any existing matting models. This is accomplished by exploring the latent space of GAN models. It is demonstrated that interpretable directions can be found in the latent space and they correspond to semantic image transformations. We further explore this property in alpha matting. Particularly, we invert an input portr…
A sketch, found in one of Leonardo da Vinci's notebooks and covered by the written notes of this genius, has been recently restored. The restoration reveals a possible self-portrait of the artist, drawn when he was young. Here, we discuss the discovery of this self-portrait and the procedure used for restoration. Actually, this is a restoration performed on the digital image of the sketch, a procedure that can easily extended and applied to ancient documents for studies of art and palaeography.
DXOMARK；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Institute of Mathematical Sciences（数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）
A Self-supervised Motion Representation for Portrait Video Generation
Zhejiang University（浙江大学）；Li Auto（利汽车）；Zhejiang Lab（浙江实验室）；Hangzhou Institute for Advanced Study,University of Chinese Academy of Sciences（杭州高等研究 institute, 中国科学院大学）
Seoul National University（首尔国立大学）；Clova AI, NAVER Corp（NAVER公司Clova AI）；Search Solutions, Inc（搜索解决方案公司）
Designing a lightweight and robust portrait segmentation algorithm is an important task for a wide range of face applications. However, the problem has been considered as a subset of the object segmentation problem. bviously, portrait segmentation has its unique requirements. First, because the portrait segmentation is performed in the middle of a whole process of many realworld applications, it requires extremely lightweight models. Second, there has not been any public datasets in this domain that contain a sufficient number of images with unbiased statistics. To solve the problems, we intro…
Significant progress has been made with artistic robots. However, existing robots fail to produce high-quality portraits in a short time. In this work, we present a drawing robot, which can automatically transfer a facial picture to a vivid portrait, and then draw it on paper within two minutes averagely. At the heart of our system is a novel portrait synthesis algorithm based on deep learning. Innovatively, we employ a self-consistency loss, which makes the algorithm capable of generating continuous and smooth brush-strokes. Besides, we propose a componential sparsity constraint to reduce the…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Apple Inc.（苹果公司）
Inverse rendering is the problem of decomposing an image into its intrinsic components, i.e. albedo, normal and lighting. To solve this ill-posed problem from single image, state-of-the-art methods in shape from shading mostly resort to supervised training on all the components on either synthetic or real datasets. Here, we propose a new self-supervised training paradigm that 1) reduces the need for full supervision on the decomposition task and 2) takes into account the relighting task. We introduce new self-supervised loss terms that leverage the consistencies between multi-lit images (image…
Seoul National University（首尔国立大学）；Clova AI, NAVER Corp（NAVER公司Clova人工智能）；NAVER LABS Europe（NAVER欧洲实验室）；Search Solutions, Inc（搜索解决方案公司）
Designing a lightweight and robust portrait segmentation algorithm is an important task for a wide range of face applications. However, the problem has been considered as a subset of the object segmentation problem and less handled in the semantic segmentation field. Obviously, portrait segmentation has its unique requirements. First, because the portrait segmentation is performed in the middle of a whole process of many real-world applications, it requires extremely lightweight models. Second, there has not been any public datasets in this domain that contain a sufficient number of images wit…
In photography, low depth of field (DOF) is an important technique to emphasize the object of interest (OOI) within an image. Thus, low DOF images are widely used in the application area of macro, portrait or sports photography. When viewing a low DOF image, the viewer implicitly concentrates on the regions that are sharper regions of the image and thus segments the image into regions of interest and non regions of interest which has a major impact on the perception of the image. Thus, a robust algorithm for the fully automatic detection of the OOI in low DOF images provides valuable informati…
Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation
University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）；Zhenguan AI Lab；Southwest Jiaotong University（西南交通大学）；Shanghai University（上海大学）；New Jersey Institute of Technology（新泽西理工学院）
Texas A&M University（得克萨斯农工大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）；Microsoft Dynamics 365 AI Research（微软Dynamics 365人工智能研究院）
Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framewo…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanjing University of Science and Technology（南京理工大学）
Cluster analysis, which focuses on the grouping and categorization of similar elements, is widely used in various fields of research. A novel and fast clustering algorithm, fission clustering algorithm, is proposed in recent year. In this article, we propose a robust fission clustering (RFC) algorithm and a self-adaptive noise identification method. The RFC and the self-adaptive noise identification method are combine to propose a self-adaptive robust fission clustering (SARFC) algorithm. Several frequently-used datasets were applied to test the performance of the proposed clustering approach …
