Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-based studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present a new MOtion FeaTure (MOFT) by eliminating content correlation information and filtering motion channels. MOFT provides a distinct set of benefits, including the ability to encode comprehensive motion information with clear interpretability, extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free video motion control framework. Our method demonstrates competitive performance in generating natural and faithful motion, providing architecture-agnostic insights and applicability in a variety of downstream tasks.
Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation
Northwestern Polytechnical University(西北工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Linköping University(林雪平大学);Australian National University(澳大利亚国立大学)
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Peking University(北京大学);Guangdong University of Technology(广东工业大学);Tencent AI Lab(腾讯AI实验室);University of Hong Kong(香港大学);Tencent PCG(腾讯平台与内容事业群)
FlowHMR: Physically Plausible Motion Capture from Video
Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学);Tencent(腾讯);East China Normal University(华东师范大学);Sun Yat-sen University(中山大学);Zhejiang University(浙江大学)
SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation
The University of Sydney, Sydney, Australia(悉尼大学);The University of Hong Kong, Hong Kong SAR, China(香港大学);The Chinese University of Hong Kong, Hong Kong SAR, China(香港中文大学)
Task-Aware Encoder Control for Deep Video Compression
Beijing Institute of Technology(北京理工大学);SenseTime Research(商汤科技研究院);The Hong Kong University of Science and Technology(香港科技大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Shanghai Jiaotong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos
Department of Electrical Engineering, Linköping University(_linköping大学电气工程系);Delft Center for Systems and Control, Delft University of Technology(代尔夫特理工大学系统与控制中心)