Traditional animation generation methods depend on training generative models with human-labelled data, entailing a sophisticated multi-stage pipeline that demands substantial human effort and incurs high training costs. Due to limited prompting plans, these methods typically produce brief, information-poor, and context-incoherent animations. To overcome these limitations and automate the animation process, we pioneer the introduction of large multimodal models (LMMs) as the core processor to build an autonomous animation-making agent, named Anim-Director. This agent mainly harnesses the advanced understanding and reasoning capabilities of LMMs and generative AI tools to create animated videos from concise narratives or simple instructions. Specifically, it operates in three main stages: Firstly, the Anim-Director generates a coherent storyline from user inputs, followed by a detailed director's script that encompasses settings of character profiles and interior/exterior descriptions, and context-coherent scene descriptions that include appearing characters, interiors or exteriors, and scene events. Secondly, we employ LMMs with the image generation tool to produce visual images of settings and scenes. These images are designed to maintain visual consistency across different scenes using a visual-language prompting method that combines scene descriptions and images of the appearing character and setting. Thirdly, scene images serve as the foundation for producing animated videos, with LMMs generating prompts to guide this process. The whole process is notably autonomous without manual intervention, as the LMMs interact seamlessly with generative tools to generate prompts, evaluate visual quality, and select the best one to optimize the final output.
Generative AI for Character Animation: A Comprehensive Survey of Techniques, Applications, and Future Directions
Computer Engineering Department, Sharif University of Technology（谢里夫理工大学计算机工程系）；Iran University of Science and Technology（伊朗科学技术大学）；Qatar Computing Research Institute（卡塔尔计算研究院）
AIOZ；University of Liverpool（利物浦大学）；VinAI Research（VinAI研究院）
Audio-driven talking head animation is a challenging research topic with many real-world applications. Recent works have focused on creating photo-realistic 2D animation, while learning different talking or singing styles remains an open problem. In this paper, we present a new method to generate talking head animation with learnable style references. Given a set of style reference frames, our framework can reconstruct 2D talking head animation based on a single input image and an audio stream. Our method first produces facial landmarks motion from the audio stream and constructs the intermedi…
One-to-All Animation: Alignment-Free Character Animation and Image Pose Transfer
Jiangnan University（江南大学）；University of Science and Technology of China（中国科学技术大学）；Chinese Academy of Sciences（中国科学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhejiang University（浙江大学）
LottieGPT: Tokenizing Vector Animation for Autoregressive Generation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；AIR, Tsinghua University（清华大学人工智能研究院）；BAAI（百度人工智能研究院）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University（南京大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
Face Animation with an Attribute-Guided Diffusion Model
Beihang University（北京航空航天大学）；Shenzhen Institutes of Advanced Technology, University of Chinese Academy of Sciences（中国科学院大学深圳先进技术研究院）；Zhongguancun Laboratory（中关村实验室）
Kyoto University（京都大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Savoie Mont Blanc（萨瓦大学-勃朗峰）
Producing realistic character animations is one of the essential tasks in human-AI interactions. Considered as a sequence of poses of a humanoid, the task can be considered as a sequence generation problem with spatiotemporal smoothness and realism constraints. Additionally, we wish to control the behavior of AI agents by giving them what to do and, more specifically, how to do it. We proposed a spatiotemporally-conditioned GAN that generates a sequence that is similar to a given sequence in terms of semantics and spatiotemporal dynamics. Using LSTM-based generator and graph ConvNet discrimina…
IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kling Team, Kuaishou Technology（快手技术 Kling 团队）；Cardiff University（卡迪夫大学）
Pose-guided person image generation and animation aim to transform a source person image to target poses. These tasks require spatial manipulation of source data. However, Convolutional Neural Networks are limited by the lack of ability to spatially transform the inputs. In this paper, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. This framework first estimates global flow fields between sources and targets. Then, corresponding local source feature patches are sampled with content-aware local attention coefficients. We show tha…
HyperMotionX: The Dataset and Benchmark with DiT-Based Pose-Guided Human Image Animation of Complex Motions
Bournemouth University（伯恩茅斯大学）；vivo BlueImage Lab, vivo Mobile Communication Co., Ltd（vivo 蓝影实验室，vivo 通信有限公司）；Peking University（北京大学）；National Cheng-Kung University（国立成功大学）
MultiAnimate: Pose-Guided Image Animation Made Extensible
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Max Planck ETH Center for Intelligent Systems（马克斯·普朗克-苏黎世联邦理工学院智能系统中心）；Technical University of Darmstadt（德累斯顿技术大学）；ETH Zürich（苏黎世联邦理工学院）；Microsoft Mixed Reality & AI Lab, UK（英国微软混合现实与人工智能实验室）
本文提出 3DiFACE，利用全卷积扩散、说话风格个性化与稀疏引导运动扩散，从单一音频生成并可关键帧编辑多样且真实的整体 3D 面部动画。
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Carnegie Mellon University（卡内基梅隆大学）
Generating smooth animations from a limited number of sequential observations has a number of applications in vision. For example, it can be used to increase number of frames per second, or generating a new trajectory only based on first and last frames, e.g. a motion of face emotions. Despite the discrete observed data (frames), the problem of generating a new trajectory is a continues problem. In addition, to be perceptually realistic, the domain of an image should not alter drastically through the trajectory of changes. In this paper, we propose a new framework, Warping Neural ODE, for gene…
3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
Disney Research（迪士尼研究院）；Rutgers University（罗格斯大学）；ETH Zurich（苏黎世联邦理工学院）
Automatically generating animation from natural language text finds application in a number of areas e.g. movie script writing, instructional videos, and public safety. However, translating natural language text into animation is a challenging task. Existing text-to-animation systems can handle only very simple sentences, which limits their applications. In this paper, we develop a text-to-animation system which is capable of handling complex sentences. We achieve this by introducing a text simplification step into the process. Building on an existing animation generation system for screenwrit…
