In this paper we present a new deep learning-driven approach to image-based synthesis of animations involving humanoid characters. Unlike previous deep approaches to image-based animation our method makes no assumptions on the type of motion to be animated nor does it require dense temporal input to produce motion. Instead we generate new animations by interpolating between user chosen keyframes, arranged sparsely in time. Utilizing a novel configuration manifold learning approach we interpolate suitable motions between these keyframes. In contrast to previous methods, ours requires less data (animations can be generated from a single youtube video) and is broadly applicable to a wide range of motions including facial motion, whole body motion and even scenes with multiple characters. These improvements serve to significantly reduce the difficulty in producing image-based animations of humanoid characters, allowing even broader audiences to express their creativity.
When animation of a humanoid figure is to be generated at run-time, instead of by replaying pre-composed motion clips, some method is required of specifying the avatar's movements in a form from which the required motion data can be automatically generated. This form must be of a more abstract nature than raw motion data: ideally, it should be independent of the particular avatar's proportions, and both writable by hand and suitable for automatic generation from higher-level descriptions of the required actions. We describe here the development and implementation of such a scripting language f…
In this paper, we present a novel method for learning to synthesize 3D mesh animation sequences with long short-term memory (LSTM) blocks and mesh-based convolutional neural networks (CNNs). Synthesizing realistic 3D mesh animation sequences is a challenging and important task in computer animation. To achieve this, researchers have long been focusing on shape analysis to develop new interpolation and extrapolation techniques. However, such techniques have limited learning capabilities and therefore can produce unrealistic animation. Deep architectures that operate directly on mesh sequences r…
We propose a novel deep learning framework for animation video resequencing. Our system produces new video sequences by minimizing a perceptual distance of images from an existing animation video clip. To measure perceptual distance, we utilize the activations of convolutional neural networks and learn a perceptual distance by training these features on a small network with data comprised of human perceptual judgments. We show that with this perceptual metric and graph-based manifold learning techniques, our framework can produce new smooth and visually appealing animation video results for a …
NVIDIA（英伟达）；Remedy Entertainment；Pinscreen；University of Southern California（南加州大学）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Aalto University（阿尔托大学）
We present a real-time deep learning framework for video-based facial performance capture -- the dense 3D tracking of an actor's face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5-10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system …
3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck ETH Center for Intelligent Systems（马克斯·普朗克-苏黎世联邦理工学院智能系统中心）；Technical University of Darmstadt（德累斯顿技术大学）；ETH Zürich（苏黎世联邦理工学院）；Microsoft Mixed Reality & AI Lab, UK（英国微软混合现实与人工智能实验室）
本文提出 3DiFACE，利用全卷积扩散、说话风格个性化与稀疏引导运动扩散，从单一音频生成并可关键帧编辑多样且真实的整体 3D 面部动画。
Beijing Film Academy（北京电影学院）；Tel-Aviv University（特拉维夫大学）；Peking University（北京大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Transferring the motion style from one animation clip to another, while preserving the motion content of the latter, has been a long-standing problem in character animation. Most existing data-driven approaches are supervised and rely on paired data, where motions with the same content are performed in different styles. In addition, these approaches are limited to transfer of styles that were seen during training. In this paper, we present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring mot…
German Research Center for Artificial Intelligence (DFKI) Saarbrücken（德国人工智能研究中心（DFKI）萨尔布吕肯）；Saarland University（萨尔大学）；Max-Planck Institute for Informatics（马克斯-普朗克计算机科学研究所）
Human motion capture data has been widely used in data-driven character animation. In order to generate realistic, natural-looking motions, most data-driven approaches require considerable efforts of pre-processing, including motion segmentation and annotation. Existing (semi-) automatic solutions either require hand-crafted features for motion segmentation or do not produce the semantic annotations required for motion synthesis and building large-scale motion databases. In addition, human labeled annotation data suffers from inter- and intra-labeler inconsistencies by design. We propose a sem…
Synthesizing human motion through learning techniques is becoming an increasingly popular approach to alleviating the requirement of new data capture to produce animations. Learning to move naturally from music, i.e., to dance, is one of the more complex motions humans often perform effortlessly. Each dance movement is unique, yet such movements maintain the core characteristics of the dance style. Most approaches addressing this problem with classical convolutional and recursive neural models undergo training and variability issues due to the non-Euclidean geometry of the motion manifold stru…
An effective 3D stepping control algorithm that is computationally fast, robust, and easy to implement is extremely important and valuable to character animation research. In this paper, we present a novel technique for generating dynamic, interactive, and controllable biped stepping motions. Our approach uses a low-dimensional physics-based model to create balanced humanoid avatars that can handle a wide variety of interactive situations, such as terrain height shifting and push exertions, while remaining upright and balanced. We accomplish this by combining the popular inverted-pendulum mode…
Deep learning typically requires vast numbers of training examples in order to be used successfully. Conversely, motion capture data is often expensive to generate, requiring specialist equipment, along with actors to generate the prescribed motions, meaning that motion capture datasets tend to be relatively small. Motion capture data does however provide a rich source of information that is becoming increasingly useful in a wide variety of applications, from gesture recognition in human-robot interaction, to data driven animation. This project illustrates how deep convolutional networks can b…
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
The field of physics-based animation is gaining importance due to the increasing demand for realism in video games and films, and has recently seen wide adoption of data-driven techniques, such as deep reinforcement learning (RL), which learn control from (human) demonstrations. While RL has shown impressive results at reproducing individual motions and interactive locomotion, existing methods are limited in their ability to generalize to new motions and their ability to compose a complex motion sequence interactively. In this paper, we propose a physics-based universal neural controller (UniC…
University of Tsukuba（筑波大学）；Toyohashi University of Technology（丰桥技术科学大学）
Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future …
