We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses. Existing pose transfer methods exhibit significant visual artifacts when applying to a novel scene, resulting in temporal inconsistency and failures in preserving the identity and textures of the person. To address these limitations, we design a compositional neural network that predicts the silhouette, garment labels, and textures. Each modular network is explicitly dedicated to a subtask that can be learned from the synthetic data. At the inference time, we utilize the trained network to produce a unified representation of appearance and its labels in UV coordinates, which remains constant across poses. The unified representation provides an incomplete yet strong guidance to generating the appearance in response to the pose change. We use the trained network to complete the appearance and render it with the background. With these strategies, we are able to synthesize human animations that can preserve the identity and appearance of the person in a temporally coherent way without any fine-tuning of the network on the testing scene. Experiments show that our method outperforms the state-of-the-arts in terms of synthesis quality, temporal coherence, and generalization ability.
SenseTime Research and Tetras.AI（商汤科技研究院与Tetras.AI）；Rutgers University（罗格斯大学）；Sun Yat-sen University（中山大学）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab Nanyang Technological University（南洋理工大学S-Lab）
In the animation industry, cartoon videos are usually produced at low frame rate since hand drawing of such frames is costly and time-consuming. Therefore, it is desirable to develop computational models that can automatically interpolate the in-between animation frames. However, existing video interpolation methods fail to produce satisfying results on animation data. Compared to natural videos, animation videos possess two unique characteristics that make frame interpolation difficult: 1) cartoons comprise lines and smooth color pieces. The smooth areas lack textures and make it difficult to…
Adobe Research（Adobe研究院）；University College London（伦敦大学学院）
Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing. While many recent methods have been proposed to tackle this problem, handling loose garments with complex textures and high dynamic motion still remains challenging. In this paper, we propose a video based appearance synthesis method that tackles such challenges and demonstrates high quality results for in-the-wild videos that have not been shown before. Specifically, we adopt a StyleGAN based architecture to the task of person specific video based motion retargeting. We i…
Face Animation with an Attribute-Guided Diffusion Model
Beihang University（北京航空航天大学）；Shenzhen Institutes of Advanced Technology, University of Chinese Academy of Sciences（中国科学院大学深圳先进技术研究院）；Zhongguancun Laboratory（中关村实验室）
RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Zhejiang University（浙江大学）；Hupan Lab（虎斑实验室）；Southern University of Science and Technology（南方科技大学）；Shenzhen University（深圳大学）
Ubisoft China AI & Data Lab（育碧中国AI与数据实验室）；Sichuan University（四川大学）；Ubisoft, Singapore（育碧新加坡）；INRIA Nancy Grand Est（法国国家信息与自动化研究所南锡大东区中心）
This work introduces a novel strategy for generating synthetic training data for 2D and 3D pose estimation of animals using keyframe animations. With the objective to automate the process of creating animations for wildlife, we train several 2D and 3D pose estimation models with synthetic data, and put in place an end-to-end pipeline called ZooBuilder. The pipeline takes as input a video of an animal in the wild, and generates the corresponding 2D and 3D coordinates for each joint of the animal's skeleton. With this approach, we produce motion capture data that can be used to create animations…
Pose-guided person image generation and animation aim to transform a source person image to target poses. These tasks require spatial manipulation of source data. However, Convolutional Neural Networks are limited by the lack of ability to spatially transform the inputs. In this paper, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. This framework first estimates global flow fields between sources and targets. Then, corresponding local source feature patches are sampled with content-aware local attention coefficients. We show tha…
University of Ontario Institute of Technology（安大略理工大学）
There is a delicate balance between automating repetitive work in creative domains while staying true to an artist's vision. The animation industry regularly outsources large animation workloads to foreign countries where labor is inexpensive and long hours are common. Automating part of this process can be incredibly useful for reducing costs and creating manageable workloads for major animation studios and outsourced artists. We present a method for automating line art colorization by keeping artists in the loop to successfully reduce this workload while staying true to an artist's vision. B…
HyperMotionX: The Dataset and Benchmark with DiT-Based Pose-Guided Human Image Animation of Complex Motions
Bournemouth University（伯恩茅斯大学）；vivo BlueImage Lab, vivo Mobile Communication Co., Ltd（vivo 蓝影实验室，vivo 通信有限公司）；Peking University（北京大学）；National Cheng-Kung University（国立成功大学）
MultiAnimate: Pose-Guided Image Animation Made Extensible
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
University of Trento（特伦托大学）；Shandong University（山东大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Texas State University（德克萨斯州立大学）；Huawei Research Ireland（华为爱尔兰研究院）
In this paper we address the problem of unsupervised gaze correction in the wild, presenting a solution that works without the need for precise annotations of the gaze angle and the head pose. We have created a new dataset called CelebAGaze, which consists of two domains X, Y, where the eyes are either staring at the camera or somewhere else. Our method consists of three novel modules: the Gaze Correction module (GCM), the Gaze Animation module (GAM), and the Pretrained Autoencoder module (PAM). Specifically, GCM and GAM separately train a dual in-painting network using data from the domain $X…
SynergyWarpNet: Attention-Guided Cooperative Warping for Neural Portrait Animation
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能联合实验室，人工智能研究院，上海交通大学）；Defense Innovation Institute, Academy of Military Science（国防创新研究院，军事科学院）；Intelligent Game and Decision Laboratory（智能游戏与决策实验室）
Image animation generates a video of a source image following the motion of a driving video. State-of-the-art self-supervised image animation approaches warp the source image according to the motion of the driving video and recover the warping artifacts by inpainting. These approaches mostly use vanilla convolution for inpainting, and vanilla convolution does not distinguish between valid and invalid pixels. As a result, visual artifacts are still noticeable after inpainting. CutMix is a state-of-the-art regularization strategy that cuts and mixes patches of images and is widely studied in dif…
CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
Technical University of Darmstadt（德累斯顿技术大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tuebingen（图宾根大学）；Tubingen AI Center（图宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck ETH Center for Learning System（马克斯·普朗克-苏黎世联邦理工学院学习系统中心）
While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displ…
This paper studies the task of conditional Human Motion Animation (cHMA). Given a source image and a driving video, the model should animate the new frame sequence, in which the person in the source image should perform a similar motion as the pose sequence from the driving video. Despite the success of Generative Adversarial Network (GANs) methods in image and video synthesis, it is still very challenging to conduct cHMA due to the difficulty in efficiently utilizing the conditional guided information such as images or poses, and generating images of good visual quality. To this end, this pap…
University of Vienna（维也纳大学）；NTNU Trondheim（挪威科技大学）
Motions of virtual characters in movies or video games are typically generated by recording actors using motion capturing methods. Animations generated this way often need postprocessing, such as improving the periodicity of cyclic animations or generating entirely new motions by interpolation of existing ones. Furthermore, search and classification of recorded motions becomes more and more important as the amount of recorded motion data grows. In this paper, we will apply methods from shape analysis to the processing of animations. More precisely, we will use the by now classical elastic metr…
University of Trento（特伦托大学）；Zhejiang University（浙江大学）；Shandong University（山东大学）；ETH Zurich（苏黎世联邦理工学院）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；Illinois Institute of Technology（伊利诺伊理工学院）
This paper proposes a gaze correction and animation method for high-resolution, unconstrained portrait images, which can be trained without the gaze angle and the head pose annotations. Common gaze-correction methods usually require annotating training data with precise gaze, and head pose information. Solving this problem using an unsupervised method remains an open problem, especially for high-resolution face images in the wild, which are not easy to annotate with gaze and head pose labels. To address this issue, we first create two new portrait datasets: CelebGaze and high-resolution CelebH…
Washington University（华盛顿大学）；Tsinghua University（清华大学）
This paper proposes an algorithm that turns a regular video capturing urban scenes into a high-quality endless animation, known as a Cinemagraph. The creation of a Cinemagraph usually requires a static camera in a carefully configured scene. The task becomes challenging for a regular video with a moving camera and objects. Our approach first warps an input video into the viewpoint of a reference camera. Based on the warped video, we propose effective temporal analysis algorithms to detect regions with static geometry and dynamic appearance, where geometric modeling is reliable and visually att…
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University（上海交通大学 USC-文化创意产业学院）；Institut Universitaire de France (IUF)（法国大学研究院）；University of Nantes（南特大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. W…
Inria Grenoble Rhône-Alpes（法国国家信息与自动化研究所格勒诺布尔-罗纳-阿尔卑斯分校）；Laboratoire Jean Kuntzmann（让·昆茨曼实验室）
This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint a…
One-Shot Learning for Pose-Guided Person Image Synthesis in the Wild
University of Electronic Science and Technology of China（电子科技大学）；China Mobile Zijin (Jiangsu) Innovation Research Institute Co., Ltd（中国移动紫金（江苏）创新研究院有限公司）；Zhejiang Sci-Tech University（浙江理工大学）；Jilin University（吉林大学）；Huawei Canada（华为加拿大公司）；Nanjing University（南京大学）；China Mobile Research Institute（中国移动研究院）
