Diffusion models have emerged as a powerful tool for generating high-quality images, videos, and 3D content. While sampling guidance techniques like CFG improve quality, they reduce diversity and motion. Autoguidance mitigates these issues but demands extra weak model training, limiting its practicality for large-scale models. In this work, we introduce Spatiotemporal Skip Guidance (STG), a simple training-free sampling guidance method for enhancing transformer-based video diffusion models. STG employs an implicit weak model via self-perturbation, avoiding the need for external models or additional training. By selectively skipping spatiotemporal layers, STG produces an aligned, degraded version of the original model to boost sample quality without compromising diversity or dynamic degree. Our contributions include: (1) introducing STG as an efficient, high-performing guidance technique for video diffusion models, (2) eliminating the need for auxiliary models by simulating a weak model through layer skipping, and (3) ensuring quality-enhanced guidance without compromising sample diversity or dynamics unlike CFG. For additional results, visit https://junhahyung.github.io/STGuidance.
GlobalPaint: Spatiotemporal Coherent Video Outpainting with Global Feature Guidance
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Microsoft Research Asia（微软亚洲研究院）；University of Science and Technology of China（中国科学技术大学）
Canadian Institute for Advanced Research（加拿大高等研究院）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；University of Guelph（圭尔夫大学）；Apple Inc（苹果公司）
Deep neural networks require collecting and annotating large amounts of data to train successfully. In order to alleviate the annotation bottleneck, we propose a novel self-supervised representation learning approach for spatiotemporal features extracted from videos. We introduce Skip-Clip, a method that utilizes temporal coherence in videos, by training a deep model for future clip order ranking conditioned on a context clip as a surrogate objective for video future prediction. We show that features learned using our method are generalizable and transfer strongly to downstream tasks. For acti…
VR-JEPA: Learning Contrastive-State Latent Guidance for Generation-based Video Reasoning
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shenzhen Loop Area Institute（深圳河套学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tencent（腾讯）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；UiT The Arctic University of Norway（挪威北极大学）
Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer
Shanghai Jiao Tong University（上海交通大学）；Xinhua Hospital Affiliated to Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属新华医院）；Rice University（莱斯大学）；University of Münster（明斯特大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta Reality Labs Research（Meta现实实验室）
Adaptive sampling that exploits the spatiotemporal redundancy in videos is critical for always-on action recognition on wearable devices with limited computing and battery resources. The commonly used fixed sampling strategy is not context-aware and may under-sample the visual content, and thus adversely impacts both computation efficiency and accuracy. Inspired by the concepts of foveal vision and pre-attentive processing from the human visual perception mechanism, we introduce a novel adaptive spatiotemporal sampling scheme for efficient action recognition. Our system pre-scans the global sc…
Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation
South China University of Technology（南方科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学研究中心，清华大学）；Department of Automation, BNRist, Tsinghua University（清华大学自动化系，北京信息科学研究中心）
Department of Information Engineering Hong Kong；Department of Information Engineering
Since the wide employment of deep learning frameworks in video salient object detection, the accuracy of the recent approaches has made stunning progress. These approaches mainly adopt the sequential modules, based on optical flow or recurrent neural network (RNN), to learn robust spatiotemporal features. These modules are effective but significantly increase the computational burden of the corresponding deep models. In this paper, to simplify the network and maintain the accuracy, we present a lightweight network tailored for video salient object detection through the spatiotemporal knowledge…
Tianjin University（天津大学）；Key Laboratory of Artificial Intelligence, Ministry of Education（教育部人工智能重点实验室）
As moving objects always draw more attention of human eyes, the temporal motive information is always exploited complementarily with spatial information to detect salient objects in videos. Although efficient tools such as optical flow have been proposed to extract temporal motive information, it often encounters difficulties when used for saliency detection due to the movement of camera or the partial movement of salient objects. In this paper, we investigate the complimentary roles of spatial and temporal information and propose a novel dynamic spatiotemporal network (DS-Net) for more effect…
PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；College of Mechanical and Vehicle Engineering, Chongqing University（重庆大学机械与车辆工程学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Latent Interpolation Learning Using Diffusion Models for Cardiac Volume Reconstruction
School of Computation, Information and Technology, Technical University Munich（技术大学慕尼黑计算信息学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（牛津大学生物医学工程研究所）；Imperial College London（伦敦帝国学院）；University of Sheffield（谢菲尔德大学）；NVIDIA（英伟达）；Department of Computing, Imperial College London（伦敦帝国学院计算机系）；Department AIBE of Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡大学AIBE系）；School of Medicine, Klinikum Rechts der Isar, Technical University of Munich（慕尼黑技术大学医学院，莱纳特医院）
Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated videos generally suffer from temporal and semantic inconsistency. To address this, many existing works adopt spatiotemporal consistency constraints incorporating temporal information based on motion estimation. However, the inaccuracies in the estimation of motion deteriorate the quality of the guidance towards spatiotemporal consistency, which leads to unstable translation. In th…
Technical University of Munich（慕尼黑工业大学）；Eskisehir Osmangazi University（埃斯基谢希尔奥桑加齐大学）
Convolutional Neural Networks with 3D kernels (3D-CNNs) currently achieve state-of-the-art results in video recognition tasks due to their supremacy in extracting spatiotemporal features within video frames. There have been many successful 3D-CNN architectures surpassing the state-of-the-art results successively. However, nearly all of them are designed to operate offline creating several serious handicaps during online operation. Firstly, conventional 3D-CNNs are not dynamic since their output features represent the complete input clip instead of the most recent frame in the clip. Secondly, t…
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
From Sight to Insight: Unleashing Eye-Tracking in Weakly Supervised Video Salient Object Detection
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究院）；School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；ByteDance China（字节跳动中国）；ByteDance USA（字节跳动美国）；Lingnan University（岭南大学）
Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence (S4) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all image-tokens equally as done by S4 model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective S4 (i.e., S5) model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term …
