We present a novel approach for image-animation of a source image by a driving video, both depicting the same type of object. We do not assume the existence of pose models and our method is able to animate arbitrary objects without the knowledge of the object's structure. Furthermore, both, the driving video and the source image are only seen during test-time. Our method is based on a shared mask generator, which separates the foreground object from its background, and captures the object's general pose and shape. To control the source of the identity of the output frame, we employ perturbations to interrupt the unwanted identity information on the driver's mask. A mask-refinement module then replaces the identity of the driver with the identity of the source. Conditioned on the source image, the transformed mask is then decoded by a multi-scale generator that renders a realistic image, in which the content of the source frame is animated by the pose in the driving video. Due to the lack of fully supervised data, we train on the task of reconstructing frames from the same video the source image is taken from. Our method is shown to greatly outperform the state-of-the-art methods on multiple benchmarks. Our code and samples are available at https://github.com/itsyoavshalev/Image-Animation-with-Perturbed-Masks.
Motion transfer is the task of synthesizing future video frames of a single source image according to the motion from a given driving video. In order to solve it, we face the challenging complexity of motion representation and the unknown relations between the driving video and the source image. Despite its difficulty, this problem attracted great interests from researches at the recent years, with gradual improvements. The goal is often thought as the decoupling of motion and appearance, which is may be solved by extracting the motion from keypoint movement. We chose to tackle the generic, un…
National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；North China University of Technology（北方工业大学）
Image matting plays an important role in image and video editing. However, the formulation of image matting is inherently ill-posed. Traditional methods usually employ interaction to deal with the image matting problem with trimaps and strokes, and cannot run on the mobile phone in real-time. In this paper, we propose a real-time automatic deep matting approach for mobile devices. By leveraging the densely connected blocks and the dilated convolution, a light full convolutional network is designed to predict a coarse binary mask for portrait images. And a feathering block, which is edge-preser…
IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kling Team, Kuaishou Technology（快手技术 Kling 团队）；Cardiff University（卡迪夫大学）
MultiAnimate: Pose-Guided Image Animation Made Extensible
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
Image animation brings life to the static object in the source image according to the driving video. Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge. However, it remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images. In this paper, a new end-to-end unsupervised motion transfer framework is proposed to overcome such issue. Firstly, we propose thin-plate spline motion estimation to produce a more flexible optical flow, w…
3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement
Engineering Research Center of Autonomous Unmanned System Technology（自主无人系统工程研究中心）；Ministry of Education（教育部）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；University of Warwick（沃林格大学）；Zhejiang Yuexiu University（浙江越秀大学）；University of Surrey（萨里大学）
EverybodyDance: Bipartite Graph-Based Identity Correspondence for Multi-Character Animation
University of Science and Technology of China（中国科学技术大学）；Li Auto；Communication University of China（通信大学）；MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC（教育部脑科学与智能感知认知重点实验室，中国科学技术大学）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure AI（微软Azure人工智能）
3D photography renders a static image into a video with appealing 3D visual effects. Existing approaches typically first conduct monocular depth estimation, then render the input frame to subsequent frames with various viewpoints, and finally use an inpainting model to fill those missing/occluded regions. The inpainting model plays a crucial role in rendering quality, but it is normally trained on out-of-domain data. To reduce the training and inference gap, we propose a novel self-supervised diffusion model as the inpainting module. Given a single input image, we automatically construct a tra…
CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation
The University of Hong Kong（香港大学）；Ant Group（蚂蚁集团）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Saarland Informatics Campus（萨尔兰信息学园区）；German Research Centre for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Max Planck Institute for Informatics (MPII)（马克斯·普朗克信息学研究所（MPII））
Zero-shot High-fidelity and Pose-controllable Character Animation
Fudan University（复旦大学）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；School of CS, Fudan University（复旦大学计算机科学技术学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；OPPO AI Center（OPPO人工智能中心）；South China University Of Technology（华南理工大学）；Westlake University（西湖大学）
BMO Lab for Creative Research（BMO创意研究实验室）；University of Toronto（多伦多大学）
Advances in the realm of Generative Adversarial Networks (GANs) have led to architectures capable of producing amazingly realistic images such as StyleGAN2, which, when trained on the FFHQ dataset, generates images of human faces from random vectors in a lower-dimensional latent space. Unfortunately, this space is entangled - translating a latent vector along its axes does not correspond to a meaningful transformation in the output space (e.g., smiling mouth, squinting eyes). The model behaves as a black box, providing neither control over its output nor insight into the structures it has lear…
We introduce a novel framework to build a model that can learn how to segment objects from a collection of images without any human annotation. Our method builds on the observation that the location of object segments can be perturbed locally relative to a given background without affecting the realism of a scene. Our approach is to first train a generative model of a layered scene. The layered representation consists of a background image, a foreground image and the mask of the foreground. A composite image is then obtained by overlaying the masked foreground image onto the background. The ge…
Intel Labs（英特尔实验室）；University of Southern California（南加州大学）
Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on either activation or weight conditioned convolution operations. However, such conditional learning costs additional multiply-accumulate (MAC) or addition operations, increasing inference memory and compute costs. To that end, we present a sparse mixture once for all adversarial training (SMART), that allows a model to train once and then in-situ trade-off between accuracy and robustness, that too at a reduced compute and parameter overhead. In particula…
Segmenting unseen object instances in cluttered environments is an important capability that robots need when functioning in unstructured environments. While previous methods have exhibited promising results, they still tend to provide incorrect results in highly cluttered scenes. We postulate that a network architecture that encodes relations between objects at a high-level can be beneficial. Thus, in this work, we propose a novel framework that refines the output of such methods by utilizing a graph-based representation of instance masks. We train deep networks capable of sampling smart pert…
Beijing Institute of Technology（北京理工大学）；University of Adelaide（阿德莱德大学）
Previous harmonization methods focus on adjusting one inharmonious region in an image based on an input mask. They may face problems when dealing with different perturbations on different semantic regions without available input masks. To deal with the problem that one image has been pasted with several foregrounds coming from different images and needs to harmonize them towards different domain directions without any mask as input, we propose a new semantic-guided multi-mask image harmonization task. Different from the previous single-mask image harmonization task, each inharmonious image is …
Structured representations such as keypoints are widely used in pose transfer, conditional image generation, animation, and 3D reconstruction. However, their supervised learning requires expensive annotation for each target domain. We propose a self-supervised method that learns to disentangle object structure from the appearance with a graph of 2D keypoints linked by straight edges. Both the keypoint location and their pairwise edge weights are learned, given only a collection of images depicting the same object class. The resulting graph is interpretable, for example, AutoLink recovers the h…
Google Research（谷歌研究院）；The Hebrew University of Jerusalem（希伯来大学）
Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtain…
Segment-to-Act: Label-Noise-Robust Action-Prompted Video Segmentation Towards Embodied Intelligence
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China（人工智能与机器人学院及机器人视觉感知与控制技术国家工程研究中心，湖南大学，中国）；Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Germany（人机学与机器人研究所，卡尔斯鲁厄理工学院，德国）
