Text-driven motion generation has achieved substantial progress with the emergence of diffusion models. However, existing methods still struggle to generate complex motion sequences that correspond to fine-grained descriptions, depicting detailed and accurate spatio-temporal actions. This lack of fine controllability limits the usage of motion generation to a larger audience. To tackle these challenges, we present FineMoGen, a diffusion-based motion generation and editing framework that can synthesize fine-grained motions, with spatial-temporal composition to the user instructions. Specifically, FineMoGen builds upon diffusion model with a novel transformer architecture dubbed Spatio-Temporal Mixture Attention (SAMI). SAMI optimizes the generation of the global attention template from two perspectives: 1) explicitly modeling the constraints of spatio-temporal composition; and 2) utilizing sparsely-activated mixture-of-experts to adaptively extract fine-grained features. To facilitate a large-scale study on this new fine-grained motion generation task, we contribute the HuMMan-MoGen dataset, which consists of 2,968 videos and 102,336 fine-grained spatio-temporal descriptions. Extensive experiments validate that FineMoGen exhibits superior motion generation quality over state-of-the-art methods. Notably, FineMoGen further enables zero-shot motion editing capabilities with the aid of modern large language models (LLM), which faithfully manipulates motion sequences with fine-grained instructions. Project Page: https://mingyuan-zhang.github.io/projects/FineMoGen.html
Cut-and-Paste: Subject-Driven Video Editing with Attention Control
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；School of Computer Science, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机学院）
IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning
School of Information and Communication Engineering, Communication University of China（中国通信大学信息与通信工程学院）；Show Lab, National University of Singapore（新加坡国立大学Show实验室）；Baidu Inc., Beijing, China（百度公司）
Google Research（谷歌研究院）；The Hebrew University of Jerusalem（希伯来大学）
Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtain…
VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing
International Artificial Intelligence Center of Morocco University Mohammed VI Polytechnic Rabat, Morocco（摩洛哥国际人工智能中心摩洛哥穆莱·伊沙克·穆莱·阿卜杜勒阿齐兹 polytechnic 大学拉巴特分校）；University Mohammed VI Polytechnic Rabat, Morocco（摩洛哥穆莱·伊沙克·穆莱·阿卜杜勒阿齐兹 polytechnic 大学拉巴特分校）；Sorbonne University, LIP6 - UMR 7606 CNRS, France（索邦大学，LIP6 - UMR 7606 CNRS，法国）
Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates
CVLab, École Polytechnique Fédérale de Lausanne (EPFL)（瑞士联邦理工学院（EPFL）计算机视觉实验室）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Pinscreen and Mohamed bin Zayed University of Artificial Intelligence（Pinscreen和马尔代夫人工智能大学）
Mining Multi-Modality Spatio-Temporal Cues for Video Important Person Identification
School of Computer Science and Technology, Wuhan University of Science and Technology（武汉科技大学计算机科学与技术学院）；Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology（湖北省智能信息处理与实时工业系统重点实验室）；School of Computer Science, National Engineering Research Center for Multimedia Software, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机科学学院，国家多媒体软件工程技术研究中心，湖北省多媒体与网络通信工程重点实验室，武汉大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
Institute of Computing Technology（计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Technologies（华为技术有限公司）；University of Science and Technology of China（中国科学技术大学）
In recent years, creative content generations like style transfer and neural photo editing have attracted more and more attention. Among these, cartoonization of real-world scenes has promising applications in entertainment and industry. Different from image translations focusing on improving the style effect of generated images, video cartoonization has additional requirements on the temporal consistency. In this paper, we propose a spatially-adaptive semantic alignment framework with perceptual motion consistency for coherent video cartoonization in an unsupervised manner. The semantic align…
ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and Key Laboratory of System Control and Information Processing, Ministry of Education of China（自动化与智能感知学院，上海交通大学；系统控制与信息处理重点实验室，中华人民共和国教育部）
SimInsert: Seamless Video Object Insertion via Regional Sparse Attention Fusion
State Key Laboratory of Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；JIUTIAN Research（中移九天）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Zhejiang Sci-Tech University（浙江科技学院）；The University of British Columbia（不列颠哥伦比亚大学）
The task of action recognition or action detection involves analyzing videos and determining what action or motion is being performed. The primary subject of these videos are predominantly humans performing some action. However, this requirement can be relaxed to generalize over other subjects such as animals or robots. The applications can range from anywhere between human-computer inter-action to automated video editing proposals. When we consider spatiotemporal action recognition, we deal with action localization. This task not only involves determining what action is being performed but al…
We introduce Emu3.5, a large-scale multimodal world model that natively predicts the next state across vision and language. Emu3.5 is pre-trained end-to-end with a unified next-token prediction objective on a corpus of vision-language interleaved data containing over 10 trillion tokens, primarily derived from sequential frames and transcripts of internet videos. The model naturally accepts interleaved vision-language inputs and generates interleaved vision-language outputs. Emu3.5 is further post-trained with large-scale reinforcement learning to enhance multimodal reasoning and generation. To…
We present an implicit neural representation to learn the spatio-temporal space of kinematic motions. Unlike previous work that represents motion as discrete sequential samples, we propose to express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF). Specifically, we use a neural network to learn this function for miscellaneous sets of motions, which is designed to be a generative model conditioned on a temporal coordinate $t$ and a random vector $z$ for controlling the style. The model is then trained as a Variational Autoencoder (VAE) with m…
