Zero-shot High-fidelity and Pose-controllable Character Animation
Fudan University（复旦大学）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；School of CS, Fudan University（复旦大学计算机科学技术学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；OPPO AI Center（OPPO人工智能中心）；South China University Of Technology（华南理工大学）；Westlake University（西湖大学）
Image-to-video (I2V) generation aims to create a video sequence from a single image, which requires high temporal coherence and visual fidelity. However, existing approaches suffer from inconsistency of character appearances and poor preservation of fine details. Moreover, they require a large amount of video data for training, which can be computationally demanding. To address these limitations, we propose PoseAnimate, a novel zero-shot I2V framework for character animation. PoseAnimate contains three key components: 1) a Pose-Aware Control Module (PACM) that incorporates diverse pose signals into text embeddings, to preserve character-independent content and maintain precise alignment of actions. 2) a Dual Consistency Attention Module (DCAM) that enhances temporal consistency and retains character identity and intricate background details. 3) a Mask-Guided Decoupling Module (MGDM) that refines distinct feature perception abilities, improving animation fidelity by decoupling the character and background. We also propose a Pose Alignment Transition Algorithm (PATA) to ensure smooth action transition. Extensive experiment results demonstrate that our approach outperforms the state-of-the-art training-based methods in terms of character consistency and detail fidelity. Moreover, it maintains a high level of temporal coherence throughout the generated animations.
Erase but Preserve: Controllable Removal of Copyrighted Animation Characters via Optimized Semantic Anchors
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Laboratory for Big Data and Decision, School of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院大数据与决策实验室）
Nanchang Hangkong University（南昌航空大学）；Dalian University of Technology（大连理工大学）；Cardiff University（卡迪夫大学）
Most of the existing audio-driven 3D facial animation methods suffered from the lack of detailed facial expression and head pose, resulting in unsatisfactory experience of human-robot interaction. In this paper, a novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. To synthesize real and detailed expression, a hierarchical decomposition strategy is proposed to encode the audio signal into both a global latent feature and a local vertex-wise control feature. Then the local and global audio features combined with vertex spati…
Consistent and Controllable Image Animation with Motion Linear Diffusion Transformers
Department of Data Science & AI, Faculty of Information Technology, Monash University（数据科学与人工智能系，信息科技学院，墨尔本大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Animating portraits using speech has received growing attention in recent years, with various creative and practical use cases. An ideal generated video should have good lip sync with the audio, natural facial expressions and head motions, and high frame quality. In this work, we present SPACE, which uses speech and a single image to generate high-resolution, and expressive videos with realistic head pose, without requiring a driving video. It uses a multi-stage approach, combining the controllability of facial landmarks with the high-quality synthesis power of a pretrained face generator. SPA…
3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement
Engineering Research Center of Autonomous Unmanned System Technology（自主无人系统工程研究中心）；Ministry of Education（教育部）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；University of Warwick（沃林格大学）；Zhejiang Yuexiu University（浙江越秀大学）；University of Surrey（萨里大学）
RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Zhejiang University（浙江大学）；Hupan Lab（虎斑实验室）；Southern University of Science and Technology（南方科技大学）；Shenzhen University（深圳大学）
Millions of images of human faces are captured every single day; but these photographs portray the likeness of an individual with a fixed pose, expression, and appearance. Portrait image animation enables the post-capture adjustment of these attributes from a single image while maintaining a photorealistic reconstruction of the subject's likeness or identity. Still, current methods for portrait image animation are typically based on 2D warping operations or manipulations of a 2D generative adversarial network (GAN) and lack explicit mechanisms to enforce multi-view consistency. Thus these meth…
DisPose: Disentangling Pose Guidance for Controllable Human Image Animation
Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）
We present a novel Animation CelebHeads dataset (AnimeCeleb) to address an animation head reenactment. Different from previous animation head datasets, we utilize 3D animation models as the controllable image samplers, which can provide a large amount of head images with their corresponding detailed pose annotations. To facilitate a data creation process, we build a semi-automatic pipeline leveraging an open 3D computer graphics software with a developed annotation system. After training with the AnimeCeleb, recent head reenactment models produce high-quality animation head reenactment results…
The recent increase in popularity of volumetric representations for scene reconstruction and novel view synthesis has put renewed focus on animating volumetric content at high visual quality and in real-time. While implicit deformation methods based on learned functions can produce impressive results, they are `black boxes' to artists and content creators, they require large amounts of training data to generalise meaningfully, and they do not produce realistic extrapolations outside the training data. In this work we solve these issues by introducing a volume deformation method which is real-t…
Warsaw University of Technology（华沙理工大学）；Microsoft（微软）；Tooploox；Jagiellonian University（雅盖隆大学）
The creation of plausible and controllable 3D human motion animations is a long-standing problem that requires a manual intervention of skilled artists. Current machine learning approaches can semi-automate the process, however, they are limited in a significant way: they can handle only a single trajectory of the expected motion that precludes fine-grained control over the output. To mitigate that issue, we reformulate the problem of future pose prediction into pose completion in space and time where multiple trajectories are represented as poses with missing joints. We show that such a frame…
This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), has enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low…
Tampere University（坦佩雷大学）；Aalto University of Technology（阿尔托理工大学）
This paper presents a generic face animator that is able to control the pose and expressions of a given face image. The animation is driven by human interpretable control signals consisting of head pose angles and the Action Unit (AU) values. The control information can be obtained from multiple sources including external driving videos and manual controls. Due to the interpretable nature of the driving signal, one can easily mix the information between multiple sources (e.g. pose from one image and expression from another) and apply selective post-production editing. The proposed face animato…
University of Chinese Academy of Sciences（中国科学院大学）
This paper proposes a novel application system for the generation of three-dimensional (3D) character animation driven by markerless human body motion capturing. The entire pipeline of the system consists of five stages: 1) the capturing of motion data using multiple cameras, 2) detection of the two-dimensional (2D) human body joints, 3) estimation of the 3D joints, 4) calculation of bone transformation matrices, and 5) generation of character animation. The main objective of this study is to generate a 3D skeleton and animation for 3D characters using multi-view images captured by ordinary ca…
Saarland Informatics Campus（萨尔兰信息学园区）；German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Max Planck Institute for Informatics (MPII)（马克斯·普朗克信息学研究所（MPII））
