The recent advent of powerful video generation models, such as Hunyuan, WanX, Veo3, and Kling, has inaugurated a new era in the field. However, the practical deployment of these models is severely impeded by their substantial computational overhead, which stems from enormous parameter counts and the iterative, multi-step sampling process required during inference. Prior research on accelerating generative models has predominantly followed two distinct trajectories: reducing the number of sampling steps (e.g., LCM, DMD, and MagicDistillation) or compressing the model size for more efficient inference (e.g., ICMD). The potential of simultaneously compressing both to create a fast and lightweight model remains an unexplored avenue. In this paper, we propose FastLightGen, an algorithm that transforms large, computationally expensive models into fast, lightweight counterparts. The core idea is to construct an optimal teacher model, one engineered to maximize student performance, within a synergistic framework for distilling both model size and inference steps. Our extensive experiments on HunyuanVideo-ATI2V and WanX-TI2V reveal that a generator using 4-step sampling and 30\% parameter pruning achieves optimal visual quality under a constrained inference budget. Furthermore, FastLightGen consistently outperforms all competing methods, establishing a new state-of-the-art in efficient video generation.
Video capture is the most extensively utilized human perception source due to its intuitively understandable nature. A desired video capture often requires multiple environmental conditions such as ample ambient-light, unobstructed space, and proper camera angle. In contrast, wireless measurements are more ubiquitous and have fewer environmental constraints. In this paper, we propose CSI2Video, a novel cross-modal method that leverages only WiFi signals from commercial devices and a source of human identity information to recover fine-grained surveillance video in a real-time manner. Specifica…
Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have proved to be a powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space of the image generator. However, successfully generating high-fidelity and high-resolution videos has yet to be seen. In this work, we investigate how to train an autoregressive latent …
CamDirector: Towards Long-Term Coherent Video Trajectory Editing
McMaster University（麦克 master 大学）；University of Toronto（多伦多大学）；The University of Hong Kong（香港大学）；McGill University（麦吉尔大学）；Concordia University（Concordia 大学）；MBZUAI
Uniform Discrete Diffusion with Metric Path for Video Generation
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化所模式识别国家实验室）；Key Laboratory of Intelligent Information Processing, ICT, CAS（中国科学院信息科技重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Tencent Hunyuan（腾讯文英）；Shanghai Jiao Tong University（上海交通大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）
Over the past few years, various tasks involving videos such as classification, description, summarization and question answering have received a lot of attention. Current models for these tasks compute an encoding of the video by treating it as a sequence of images and going over every image in the sequence. However, for longer videos this is very time consuming. In this paper, we focus on the task of video classification and aim to reduce the computational time by using the idea of distillation. Specifically, we first train a teacher network which looks at all the frames in a video and compu…
Google Brain（谷歌大脑）；University of Michigan（密歇根大学）；University College London（伦敦大学学院）
We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked t…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Robert Bosch Centre for Data Science and AI（罗伯特·博世数据科学与人工智能中心）；NVIDIA Bangalore（英伟达班加罗尔分部）
Recently,there has been a lot of interest in building compact models for video classification which have a small memory footprint (<1 GB). While these models are compact, they typically operate by repeated application of a small weight matrix to all the frames in a video. E.g. recurrent neural network based methods compute a hidden state for every frame of the video using a recurrent weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD, have a learnable clustering matrix which is used to assign soft-clusters to every frame in the video. Since these models look at every…
Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models
Guilin University of Electronic Technology（桂林电子科技大学）；The Ohio State University（俄亥俄州立大学）；University of Arizona（亚利桑那大学）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing（西蒙斯理论计算研究所）
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology（湖北大数据安全工程研究中心，网络安全学院，华中科技大学）；Peking University（北京大学）；Henan University（河南大学）；Dalian University of Technology（大连理工大学）；Sichuan University（四川大学）；Shenzhen University（深圳大学）；Huazhong University of Science and Technology（华中科技大学）
University of Washington（华盛顿大学）；Facebook Inc.（脸书公司）
Video transmission applications (e.g., conferencing) are gaining momentum, especially in times of global health pandemic. Video signals are transmitted over lossy channels, resulting in low-quality received signals. To restore videos on recipient edge devices in real-time, we introduce an efficient video restoration network, EVRNet. EVRNet efficiently allocates parameters inside the network using alignment, differential, and fusion modules. With extensive experiments on video restoration tasks (deblocking, denoising, and super-resolution), we demonstrate that EVRNet delivers competitive perfor…
Neural fields have emerged as a powerful paradigm for representing various signals, including videos. However, research on improving the parameter efficiency of neural fields is still in its early stages. Even though neural fields that map coordinates to colors can be used to encode video signals, this scheme does not exploit the spatial and temporal redundancy of video signals. Inspired by standard video compression algorithms, we propose a neural field architecture for representing and compressing videos that deliberately removes data redundancy through the use of motion information across v…
Video object detection is a fundamental yet challenging task in computer vision. One practical solution is to take advantage of temporal information from the video and apply feature aggregation to enhance the object features in each frame. Though effective, those existing methods always suffer from low inference speeds because they use a fixed number of frames for feature aggregation regardless of the input frame. Therefore, this paper aims to improve the inference speed of the current feature aggregation-based video object detectors while maintaining their performance. To achieve this goal, w…
Recently, the image-wise implicit neural representation of videos, NeRV, has gained popularity for its promising results and swift speed compared to regular pixel-wise implicit representations. However, the redundant parameters within the network structure can cause a large model size when scaling up for desirable performance. The key reason of this phenomenon is the coupled formulation of NeRV, which outputs the spatial and temporal information of video frames directly from the frame index input. In this paper, we propose E-NeRV, which dramatically expedites NeRV by decomposing the image-wise…
StepAL: Step-aware Active Learning for Cataract Surgical Videos
Johns Hopkins University（约翰霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Malone Center for Engineering in Healthcare（马洛中心医疗工程中心）
Georgia Institute of Technology（佐治亚理工学院）；Salesforce Research（赛富时研究院）；University of Maryland（马里兰大学）
We present AdaFrame, a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame contains a Long Short-Term Memory network augmented with a global memory that provides context information for searching which frames to use over time. Trained with policy gradient methods, AdaFrame generates a prediction, determines which frame to observe next, and computes the utility, i.e., expected future rewards, of seeing more frames at each time step. At testing time, AdaFrame exploits predicted utilities to achieve adaptive lookahead inference such that the…
Row-Column Separated Attention Based Low-Light Image/Video Enhancement
School of Artificial Intelligence, Jilin University, China（吉林大学人工智能学院）；College of Software, Jilin University, China（吉林大学软件学院）；Institute of Computing Technology, Chinese Academy of Sciences, China（中国科学院计算技术研究所）
Motion-based video frame interpolation (VFI) methods have made remarkable progress with the development of deep convolutional networks over the past years. While their performance is often jeopardized by the inaccuracy of flow map estimation, especially in the case of large motion and occlusion. In this paper, we propose a flow guidance deformable compensation network (FGDCN) to overcome the drawbacks of existing motion-based methods. FGDCN decomposes the frame sampling process into two steps: a flow step and a deformation step. Specifically, the flow step utilizes a coarse-to-fine flow estima…
The Chinese University of Hong Kong（香港中文大学）；University of Surrey（萨里大学）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understa…
