Producing realistic character animations is one of the essential tasks in human-AI interactions. Considered as a sequence of poses of a humanoid, the task can be considered as a sequence generation problem with spatiotemporal smoothness and realism constraints. Additionally, we wish to control the behavior of AI agents by giving them what to do and, more specifically, how to do it. We proposed a spatiotemporally-conditioned GAN that generates a sequence that is similar to a given sequence in terms of semantics and spatiotemporal dynamics. Using LSTM-based generator and graph ConvNet discriminator, this system is trained end-to-end on a large gathered dataset of gestures, expressions, and actions. Experiments showed that compared to traditional conditional GAN, our method creates plausible, realistic, and semantically relevant humanoid animation sequences that match user expectations.
Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；The University of New South Wales（新南威尔士大学）
StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
State Key Laboratory of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Ant Group（蚂蚁集团）；College of Computer Science, Zhejiang University（浙江大学计算机学院）
3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
We present an implicit neural representation to learn the spatio-temporal space of kinematic motions. Unlike previous work that represents motion as discrete sequential samples, we propose to express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF). Specifically, we use a neural network to learn this function for miscellaneous sets of motions, which is designed to be a generative model conditioned on a temporal coordinate $t$ and a random vector $z$ for controlling the style. The model is then trained as a Variational Autoencoder (VAE) with m…
Automatic animation line art colorization is a challenging computer vision problem, since the information of the line art is highly sparse and abstracted and there exists a strict requirement for the color and style consistency between frames. Recently, a lot of Generative Adversarial Network (GAN) based image-to-image translation methods for single line art colorization have emerged. They can generate perceptually appealing results conditioned on line art images. However, these methods can not be adopted for the purpose of animation colorization because there is a lack of consideration of the…
StableAnimator: High-Quality Identity-Preserving Human Image Animation
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（复旦大学计算机学院上海市智能信息处理重点实验室）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Microsoft Research Asia（微软亚洲研究院）；Huya Inc.（虎牙公司）；Carnegie Mellon University（卡内基梅隆大学）
As the quality of few shot facial animation from landmarks increases, new applications become possible, such as ultra low bandwidth video chat compression with a high degree of realism. However, there are some important challenges to tackle in order to improve the experience in real world conditions. In particular, the current approaches fail to represent profile views without distortions, while running in a low compute regime. We focus on this key problem by introducing a multi-frames embedding dubbed Frontalizer to improve profile views rendering. In addition to this core improvement, we exp…
University of Southern California（南加州大学）；Adobe Research（奥多比研究院）；USC Institute for Creative Technologies（南加州大学创意技术研究所）；Pinscreen（品屏公司）
The ability to generate complex and realistic human body animations at scale, while following specific artistic constraints, has been a fundamental goal for the game and animation industry for decades. Popular techniques include key-framing, physics-based simulation, and database methods via motion graphs. Recently, motion generators based on deep learning have been introduced. Although these learning models can automatically generate highly intricate stylized motions of arbitrary length, they still lack user control. To this end, we introduce the problem of long-term inbetweening, which invol…
University of Hamburg（汉堡大学）；Adobe Research（Adobe 研究院）
We introduce CharacterGAN, a generative model that can be trained on only a few samples (8 - 15) of a given character. Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation. Since we only have very limited training samples, one of the key challenges lies in how to address (dis)occlusions, e.g. when a hand moves behind or in front of a body. To address this, we introduce a novel layering approach which explicitly splits the input keypoints into different layers which are…
Synthesizing human motion through learning techniques is becoming an increasingly popular approach to alleviating the requirement of new data capture to produce animations. Learning to move naturally from music, i.e., to dance, is one of the more complex motions humans often perform effortlessly. Each dance movement is unique, yet such movements maintain the core characteristics of the dance style. Most approaches addressing this problem with classical convolutional and recursive neural models undergo training and variability issues due to the non-Euclidean geometry of the motion manifold stru…
CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation
The University of Hong Kong（香港大学）；Ant Group（蚂蚁集团）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
Technical University of Darmstadt（德累斯顿技术大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tuebingen（图宾根大学）；Tubingen AI Center（图宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
