Video editing has recently achieved remarkable progress with diffusion-based generative models, enabling diverse object-level manipulations from natural language instructions. However, existing methods often struggle under occlusion, viewpoint changes, and fast object motion, where unreliable visual observations lead to inaccurate localization, temporal flickering, and inconsistent edits. In this work, we identify the absence of reliable visual anchors as a fundamental bottleneck in occlusion-robust video editing. To address this issue, we propose an occlusion-aware physics-semantic keyframe selection framework that automatically identifies an optimal anchor frame for downstream editing. Specifically, our method evaluates candidate frames from three complementary perspectives: structural completeness for avoiding truncated observations, cycle-consistent tracking stability for measuring physical reliability, and vision-language-based attribute visibility for ensuring semantic clarity. The selected keyframe is then propagated through bidirectional tracking to generate dense spatiotemporal masks, which are used as auxiliary supervision for a diffusion-based video editing backbone. By transforming occlusion handling from explicit reconstruction into reliable anchor selection, our framework enables precise and temporally consistent editing without requiring manual annotations. Extensive experiments on challenging video editing benchmarks demonstrate the effectiveness and high-quality performance of our method.
Many video editing tasks such as rotoscoping or object removal require the propagation of context across frames. While transformers and other attention-based approaches that aggregate features globally have demonstrated great success at propagating object masks from keyframes to the whole video, they struggle to propagate high-frequency details such as textures faithfully. We hypothesize that this is due to an inherent bias of global attention towards low-frequency features. To overcome this limitation, we present a two-stream approach, where high-frequency features interact locally and low-fr…
SketchVideo: Sketch-based Video Generation and Editing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）；Kuaishou Technology（快手科技）
Dynamic Motion Blending for Versatile Motion Editing
Institute for AI, Peking University（人工智能研究院，北京大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；Yuanpei College, Peking University（元培学院，北京大学）
Temporal consistency is essential for video editing applications. Existing work on layered representation of videos allows propagating edits consistently to each frame. These methods, however, can only edit object appearance rather than object shape changes due to the limitation of using a fixed UV mapping field for texture atlas. We present a shape-aware, text-driven video editing method to tackle this challenge. To handle shape changes in video editing, we first propagate the deformation field between the input and edited keyframe to all frames. We then leverage a pre-trained text-conditione…
BooM-VVT: Boosting Mask-Free Video Virtual Try-On with Image-Level Pseudo Data
Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Meituan（美团）
Person image synthesis, e.g., pose transfer, is a challenging problem due to large variation and occlusion. Existing methods have difficulties predicting reasonable invisible regions and fail to decouple the shape and style of clothing, which limits their applications on person image editing. In this paper, we propose PISE, a novel two-stage generative model for Person Image Synthesis and Editing, which is able to generate realistic person images with desired poses, textures, or semantic layouts. For human pose transfer, we first synthesize a human parsing map aligned with the target pose to r…
Catalyst4D: High-Fidelity 3D-to-4D Scene Editing via Dynamic Propagation
State Key Laboratory of Complex and Critical Software Environment, Beihang University（北京航空航天大学复杂关键软件环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Zhejiang Industrial Big Data and Robot Intelligent System Key Laboratory, Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院浙江省工业大数据与机器人智能系统重点实验室）
Hong Kong University of Science and Technology（香港科技大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；International Digital Economy Academy (IDEA)（国际数字经济研究院（IDEA））
3D GAN inversion aims to achieve high reconstruction fidelity and reasonable 3D geometry simultaneously from a single image input. However, existing 3D GAN inversion methods rely on time-consuming optimization for each individual case. In this work, we introduce a novel encoder-based inversion framework based on EG3D, one of the most widely-used 3D GAN models. We leverage the inherent properties of EG3D's latent space to design a discriminator and a background depth regularization. This enables us to train a geometry-aware encoder capable of converting the input image into corresponding latent…
3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Mixed Reality & AI Lab（微软混合现实与人工智能实验室）；Technical University of Darmstadt（达姆施塔特工业大学）
The Chinese University of Hong Kong（香港中文大学）；Adobe System Inc（奥多比系统公司）
Light fields are 4D scene representation typically structured as arrays of views, or several directional samples per pixel in a single view. This highly correlated structure is not very efficient to transmit and manipulate (especially for editing), though. To tackle these problems, we present a novel compact and editable light field representation, consisting of a set of visual channels (i.e. the central RGB view) and a complementary meta channel that encodes the residual geometric and appearance information. The visual channels in this representation can be edited using existing 2D image edit…
MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation
Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）；Show Lab, National University of Singapore（Show实验室，新加坡国立大学）
Weizmann Institute of Science（魏茨曼科学研究所）；Adobe Research（奥多比研究院）
We present a method that decomposes, or "unwraps", an input video into a set of layered 2D atlases, each providing a unified representation of the appearance of an object (or background) over the video. For each pixel in the video, our method estimates its corresponding 2D coordinate in each of the atlases, giving us a consistent parameterization of the video, along with an associated alpha (opacity) value. Importantly, we design our atlases to be interpretable and semantic, which facilitates easy and intuitive editing in the atlas domain, with minimal manual work required. Edits applied to a …
Numerous pose-guided human editing methods have been explored by the vision community due to their extensive practical applications. However, most of these methods still use an image-to-image formulation in which a single image is given as input to produce an edited image as output. This objective becomes ill-defined in cases when the target pose differs significantly from the input pose. Existing methods then resort to in-painting or style transfer to handle occlusions and preserve content. In this paper, we explore the utilization of multiple views to minimize the issue of missing informatio…
Existing GAN inversion and editing methods work well for aligned objects with a clean background, such as portraits and animal faces, but often struggle for more difficult categories with complex scene layouts and object occlusions, such as cars, animals, and outdoor images. We propose a new method to invert and edit such complex images in the latent space of GANs, such as StyleGAN2. Our key idea is to explore inversion with a collection of layers, spatially adapting the inversion process to the difficulty of the image. We learn to predict the "invertibility" of different image segments and pr…
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Meituan New Laboratory of Pattern Recognition (NLPR), CASIA（美团模式识别新实验室（NLPR），中国科学院自动化所）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Meituan Division of AMC and Department of ECE, HKUST（美团AMC部门和香港科技大学电子与计算机工程系）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences（美团人工智能学院，中国科学院大学）；Meituan（美团）
Hong Kong University of Science and Technology（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；City University of Hong Kong（香港城市大学）
We propose a novel framework to produce cartoon videos by fetching the color information from two input keyframes while following the animated motion guided by a user sketch. The key idea of the proposed approach is to estimate the dense cross-domain correspondence between the sketch and cartoon video frames, and employ a blending module with occlusion estimation to synthesize the middle frame guided by the sketch. After that, the input frames and the synthetic frame equipped with established correspondence are fed into an arbitrary-time frame interpolation pipeline to generate and refine addi…
