Deep learning typically requires vast numbers of training examples in order to be used successfully. Conversely, motion capture data is often expensive to generate, requiring specialist equipment, along with actors to generate the prescribed motions, meaning that motion capture datasets tend to be relatively small. Motion capture data does however provide a rich source of information that is becoming increasingly useful in a wide variety of applications, from gesture recognition in human-robot interaction, to data driven animation.
This project illustrates how deep convolutional networks can be used, alongside specialized data augmentation techniques, on a small motion capture dataset to learn detailed information from sequences of a specific type of motion (object transport). The project shows how these same augmentation techniques can be scaled up for use in the more complex task of motion synthesis.
By exploring recent developments in the concept of Generative Adversarial Models (GANs), specifically the Wasserstein GAN, this project outlines a model that is able to successfully generate lifelike object transportation motions, with the generated samples displaying varying styles and transport strategies.
Beijing University of Posts and Telecommunications（北京邮电大学）
Face photo-sketch synthesis and recognition has many applications in digital entertainment and law enforcement. Recently, generative adversarial networks (GANs) based methods have significantly improved the quality of image synthesis, but they have not explicitly considered the purpose of recognition. In this paper, we first propose an Identity-Aware CycleGAN (IACycleGAN) model that applies a new perceptual loss to supervise the image generation network. It improves CycleGAN on photo-sketch synthesis by paying more attention to the synthesis of key facial regions, such as eyes and nose, which …
Traffic light recognition, as a critical component of the perception module of self-driving vehicles, plays a vital role in the intelligent transportation systems. The prevalent deep learning based traffic light recognition methods heavily hinge on the large quantity and rich diversity of training data. However, it is quite challenging to collect data in various rare scenarios such as flashing, blackout or extreme weather, thus resulting in the imbalanced distribution of training data and consequently the degraded performance in recognizing rare classes. In this paper, we seek to improve traff…
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
Micro-expressions are reflections of people's true feelings and motives, which attract an increasing number of researchers into the study of automatic facial micro-expression recognition. The short detection window, the subtle facial muscle movements, and the limited training samples make micro-expression recognition challenging. To this end, we propose a novel Identity-aware and Capsule-Enhanced Generative Adversarial Network with graph-based reasoning (ICE-GAN), introducing micro-expression synthesis as an auxiliary task to assist recognition. The generator produces synthetic faces with cont…
Learning light-weight yet expressive deep networks in both image synthesis and image recognition remains a challenging problem. Inspired by a more recent observation that it is the data-specificity that makes the multi-head self-attention (MHSA) in the Transformer model so powerful, this paper proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity, as a convolutional alternative of the MHSA, while retaining the efficiency of SE and the inductive basis of convolution. It presents two designs of spatially-adapt…
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Center for Research on Intelligent Perception and Computing, CASIA（中国科学院自动化研究所智能感知与计算研究中心）；Center for Excellence in Brain Science and Intelligence Technology, CAS（中国科学院脑科学与智能技术卓越创新中心）
Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Center for Research on Intelligent Perception and Computing, CASIA（中国科学院自动化研究所智能感知与计算研究中心）；University of Chinese Academy of Sciences（中国科学院大学）
A common yet challenging scenario in periocular biometrics is cross-spectral matching - in particular, the matching of visible wavelength against near-infrared (NIR) periocular images. We propose a novel approach to cross-spectral periocular verification that primarily focuses on learning a mapping from visible and NIR periocular images to a shared latent representational subspace, and supports this effort by simultaneously learning intra-spectral image reconstruction. We show the auxiliary image reconstruction task (and in particular the reconstruction of high-level, semantic features) result…
To detect bias in face recognition networks, it can be useful to probe a network under test using samples in which only specific attributes vary in some controlled way. However, capturing a sufficiently large dataset with specific control over the attributes of interest is difficult. In this work, we describe a simulator that applies specific head pose and facial expression adjustments to images of previously unseen people. The simulator first fits a 3D morphable model to a provided image, applies the desired head pose and facial expression controls, then renders the model into an image. Next,…
Recent neural rendering methods have demonstrated accurate view interpolation by predicting volumetric density and color with a neural network. Although such volumetric representations can be supervised on static and dynamic scenes, existing methods implicitly bake the complete scene light transport into a single neural network for a given scene, including surface modeling, bidirectional scattering distribution functions, and indirect lighting effects. In contrast to traditional rendering pipelines, this prohibits changing surface reflectance, illumination, or composing other objects in the sc…
WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models
Luleå University of Technology（吕勒奥理工大学）；National Technical University of Athens（雅典国立技术大学）；Friedrich-Alexander-Universität（弗里德里希-亚历山大大学）；University of West Attica（西阿提卡大学）；University of Ioannina（约阿尼纳大学）
Palmprint De-Identification Using Diffusion Model for High-Quality and Diverse Synthesis
PAMI Research Group, Department of Computer and Information Science, University of Macau（PAMI研究组，计算机与信息科学系，澳门大学）；School of Electrical and Electronic Engineering, College of Engineering, Yonsei University（电气电子工程学院，延世大学）；School of Software, Nanchang Hangkong University（软件学院，南昌航空大学）；College of Computer Science, Sichuan University（计算机科学学院，四川大学）
Multi-task learning is an effective learning strategy for deep-learning-based facial expression recognition tasks. However, most existing methods take into limited consideration the feature selection, when transferring information between different tasks, which may lead to task interference when training the multi-task networks. To address this problem, we propose a novel selective feature-sharing method, and establish a multi-task network for facial expression recognition and facial expression synthesis. The proposed method can effectively transfer beneficial features between different tasks,…
Although significant advances have been made in face recognition (FR), FR in unconstrained environments remains challenging due to the domain gap between the semi-constrained training datasets and unconstrained testing scenarios. To address this problem, we propose a controllable face synthesis model (CFSM) that can mimic the distribution of target datasets in a style latent space. CFSM learns a linear subspace with orthogonal bases in the style latent space with precise control over the diversity and degree of synthesis. Furthermore, the pre-trained synthesis model can be guided by the FR mod…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
In the application of face recognition, eyeglasses could significantly degrade the recognition accuracy. A feasible method is to collect large-scale face images with eyeglasses for training deep learning methods. However, it is difficult to collect the images with and without glasses of the same identity, so that it is difficult to optimize the intra-variations caused by eyeglasses. In this paper, we propose to address this problem in a virtual synthesis manner. The high-fidelity face images with eyeglasses are synthesized based on 3D face model and 3D eyeglasses. Models based on deep learning…
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（奥多比研究院）
Human portraits exhibit various appearances when observed from different views under different lighting conditions. We can easily imagine how the face will look like in another setup, but computer algorithms still fail on this problem given limited observations. To this end, we present a system for portrait view synthesis and relighting: given multiple portraits, we use a neural network to predict the light-transport field in 3D space, and from the predicted Neural Light-transport Field (NeLF) produce a portrait from a new camera view under a new environmental lighting. Our system is trained o…
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We propose a novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and a discriminative model, in a way that facilitates knowledge to flow across tasks…
Fudan University（复旦大学）；Shanghai Center for Brain Science and Brain-inspired Technology（上海脑科学与类脑研究中心）
To minimize the effects of age variation in face recognition, previous work either extracts identity-related discriminative features by minimizing the correlation between identity- and age-related features, called age-invariant face recognition (AIFR), or removes age variation by transforming the faces of different age groups into the same age group, called face age synthesis (FAS); however, the former lacks visual results for model interpretation while the latter suffers from artifacts compromising downstream recognition. Therefore, this paper proposes a unified, multi-task framework to joint…
Tencent Youtu Lab（腾讯优图实验室）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Hefei University of Technology（合肥工业大学）；Shanghai Jiaotong University（上海交通大学）
Palmprints are private and stable information for biometric recognition. In the deep learning era, the development of palmprint recognition is limited by the lack of sufficient training data. In this paper, by observing that palmar creases are the key information to deep-learning-based palmprint recognition, we propose to synthesize training data by manipulating palmar creases. Concretely, we introduce an intuitive geometric model which represents palmar creases with parameterized Bézier curves. By randomly sampling Bézier parameters, we can synthesize massive training samples of diverse ident…
Fudan University（复旦大学）；Shanghai Center for Brain Science and Brain-inspired Technology（上海脑科学与类脑研究中心）
To minimize the impact of age variation on face recognition, age-invariant face recognition (AIFR) extracts identity-related discriminative features by minimizing the correlation between identity- and age-related features while face age synthesis (FAS) eliminates age variation by converting the faces in different age groups to the same group. However, AIFR lacks visual results for model interpretation and FAS compromises downstream recognition due to artifacts. Therefore, we propose a unified, multi-task framework to jointly handle these two tasks, termed MTLFace, which can learn the age-invar…
MPI Informatik（马克斯·普朗克信息学研究所）；Technical University of Denmark（丹麦技术大学）；University College London（伦敦大学学院）
We tackle the problem of generating novel-view images from collections of 2D images showing refractive and reflective objects. Current solutions assume opaque or transparent light transport along straight paths following the emission-absorption model. Instead, we optimize for a field of 3D-varying Index of Refraction (IoR) and trace light through it that bends toward the spatial gradients of said IoR according to the laws of eikonal light transport.
This research features a deep-learning based framework to address the problem of matching a given face sketch image against a face photo database. The problem of photo-sketch matching is challenging because 1) there is large modality gap between photo and sketch, and 2) the number of paired training samples is insufficient to train deep learning based networks. To circumvent the problem of large modality gap, our approach is to use an intermediate latent space between the two modalities. We effectively align the distributions of the two modalities in this latent space by employing a bidirectio…
Massachusetts Institute of Technology（麻省理工学院）；Google（谷歌）；University of California, San Diego（加州大学圣地亚哥分校）
The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular…
From Zero-shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis
University of Sheffield（谢菲尔德大学）；University of East Anglia（东英吉利大学）；University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）；Northumbria University（诺森比亚大学）
Look Across Elapse: Disentangled Representation Learning and Photorealistic Cross-Age Face Synthesis for Age-Invariant Face Recognition
National University of Singapore（新加坡国立大学）；National University of Defense Technology（中国人民解放军国防科技大学）；Panasonic R&D Center Singapore（松下新加坡研发中心）；Nanyang Technological University（南洋理工大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Qihoo 360 AI Institute（奇虎360人工智能研究院）
In this paper, we aim to address the problem of heterogeneous or cross-spectral face recognition using machine learning to synthesize visual spectrum face from infrared images. The synthesis of visual-band face images allows for more optimal extraction of facial features to be used for face identification and/or verification. We explore the ability to use Generative Adversarial Networks (GANs) for face image synthesis, and examine the performance of these images using pre-trained Convolutional Neural Networks (CNNs). The features extracted using CNNs are applied in face identification and veri…
FreeTumor: Large-Scale Generative Tumor Synthesis in Computed Tomography Images for Improving Tumor Recognition
The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯AI实验室）；Harvard University（哈佛大学）；The Chinese University of Hong Kong（香港中文大学）；The Third Affiliated Hospital of Southern Medical University（南方医科大学第三附属医院）；Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University（中山大学孙逸仙纪念医院）；The University of Hong Kong（香港大学）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港产学研协同创新研究院）
TextSSR: Diffusion-based Data Synthesis for Scene Text Recognition
College of Computer Science and Artificial Intelligence, Fudan University, China（计算机科学与人工智能学院，复旦大学）；Institute of Trustworthy Embodied AI, Fudan University, China（可信具身人工智能研究院，复旦大学）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Heterogeneous Face Recognition (HFR) aims to match faces across different domains (e.g., visible to near-infrared images), which has been widely applied in authentication and forensics scenarios. However, HFR is a challenging problem because of the large cross-domain discrepancy, limited heterogeneous data pairs, and large variation of facial attributes. To address these challenges, we propose a new HFR method from the perspective of heterogeneous data augmentation, named Face Synthesis with Identity-Attribute Disentanglement (FSIAD). Firstly, the identity-attribute disentanglement (IAD) decou…
Identity-invariant facial expression recognition (FER) has been one of the challenging computer vision tasks. Since conventional FER schemes do not explicitly address the inter-identity variation of facial expressions, their neural network models still operate depending on facial identity. This paper proposes to quantify the inter-identity variation by utilizing pairs of similar expressions explored through a specific matching process. We formulate the identity matching process as an Optimal Transport (OT) problem. Specifically, to find pairs of similar expressions from different identities, w…
Human motion is a key function to communicate information. In the application, short-form mobile video is so popular all over the world such as Tik Tok. The users would like to add more VFX so as to pursue creativity and personlity. Many special effects are added on the short video platform. These gives the users more possibility to show off these personality. The common and traditional way is to create the template of VFX. However, in order to synthesis the perfect, the users have to tedious attempt to grasp the timing and rhythm of new templates. It is not easy-to-use especially for the mobi…
Adaptive Adversarial Augmentation for Controllable Face Synthesis
Manipal University Jaipur（斋浦尔马尼帕尔大学）；Birla Institute of Technology and Science Pilani Dubai Campus（Pilani贝拉理工学院迪拜校区）；Indian Institute of Technology Jodhpur（焦特布尔印度理工学院）
Unsupervised domain adaptation is one of the challenging problems in computer vision. This paper presents a novel approach to unsupervised domain adaptations based on the optimal transport-based distance. Our approach allows aligning target and source domains without the requirement of meaningful metrics across domains. In addition, the proposal can associate the correct mapping between source and target domains and guarantee a constraint of topology between source and target domains. The proposed method is evaluated on different datasets in various problems, i.e. (i) digit recognition on MNIS…
