我们提出了Stable Video 4D 2.0（SV4D 2.0），一种用于动态3D资产生成的多视角视频扩散模型。与其前身SV4D相比，SV4D 2.0对遮挡和大运动更加鲁棒，对真实世界视频的泛化能力更强，并在细节清晰度和时空一致性方面产生更高质量的输出。我们通过在多个方面引入关键改进来实现这一目标：1）网络架构：消除对参考多视角的依赖，并设计3D和帧注意力的混合机制；2）数据：提高训练数据的质量和数量；3）训练策略：采用渐进式3D-4D训练以增强泛化能力；4）4D优化：通过两阶段细化和渐进帧采样来处理3D不一致和大运动。大量实验表明，SV4D 2.0在视觉和定量方面均取得了显著的性能提升，在新视角视频合成中实现了更好的细节（LPIPS降低14%）和4D一致性（FV4D降低44%），在4D优化中实现了LPIPS降低12%和FV4D降低24%，相比SV4D均有显著改善。项目页面：https://sv4d20.github.io。
英文摘要：
We present Stable Video 4D 2.0 (SV4D 2.0), a multi-view video diffusion model for dynamic 3D asset generation. Compared to its predecessor SV4D, SV4D 2.0 is more robust to occlusions and large motion, generalizes better to real-world videos, and produces higher-quality outputs in terms of detail sharpness and spatio-temporal consistency. We achieve this by introducing key improvements in multiple aspects: 1) network architecture: eliminating the dependency of reference multi-views and designing blending mechanism for 3D and frame attention, 2) data: enhancing quality and quantity of training data, 3) training strategy: adopting progressive 3D-4D training for better generalization, and 4) 4D optimization: handling 3D inconsistency and large motion via 2-stage refinement and progressive frame sampling. Extensive experiments demonstrate significant performance gain by SV4D 2.0 both visually and quantitatively, achieving better detail (-14\% LPIPS) and 4D consistency (-44\% FV4D) in novel-view video synthesis and 4D optimization (-12\% LPIPS and -24\% FV4D) compared to SV4D. Project page: https://sv4d20.github.io.
A Survey: Spatiotemporal Consistency in Video Generation
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）
Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Hubei Key Laboratory of Smart Internet, Huazhong University of Science and Technology（华中科技大学智慧互联网湖北省重点实验室）
ST-Gen4D: Embedding 4D Spatiotemporal Cognition into World Model for 4D Generation
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Computing, Macquarie University（麦考瑞大学计算机学院）
Enhancing Low Dose Computed Tomography Images Using Consistency Training Techniques
University of Kentucky（肯塔基大学）；Institute for Biomedical Informatics（生物医学信息学研究所）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Self-supervised learning has shown great potentials in improving the deep learning model in an unsupervised manner by constructing surrogate supervision signals directly from the unlabeled data. Different from existing works, we present a novel way to obtain the surrogate supervision signal based on high-level feature maps under consistency regularization. In this paper, we propose a Spatio-Temporal Consistency Regularization between different output features generated from a siamese network including a clean path fed with original video and a noise path fed with the corresponding augmented vi…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Google（谷歌）
We leverage unsupervised learning of depth, egomotion, and camera intrinsics to improve the performance of single-image semantic segmentation, by enforcing 3D-geometric and temporal consistency of segmentation masks across video frames. The predicted depth, egomotion, and camera intrinsics are used to provide an additional supervision signal to the segmentation model, significantly enhancing its quality, or, alternatively, reducing the number of labels the segmentation model needs. Our experiments were performed on the ScanNet dataset.
Spatio-Temporal video grounding (STVG) focuses on retrieving the spatio-temporal tube of a specific object depicted by a free-form textual expression. Existing approaches mainly treat this complicated task as a parallel frame-grounding problem and thus suffer from two types of inconsistency drawbacks: feature alignment inconsistency and prediction inconsistency. In this paper, we present an end-to-end one-stage framework, termed Spatio-Temporal Consistency-Aware Transformer (STCAT), to alleviate these issues. Specially, we introduce a novel multi-modal template as the global objective to addre…
VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Innovation Institute（上海创新研究院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
Hunan Normal University（湖南师范大学）；Meta Reality Labs（Meta现实实验室）；Wuhan University（武汉大学）；Lakehead University（湖首大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
It is challenging to annotate large-scale datasets for supervised video shadow detection methods. Using a model trained on labeled images to the video frames directly may lead to high generalization error and temporal inconsistent results. In this paper, we address these challenges by proposing a Spatio-Temporal Interpolation Consistency Training (STICT) framework to rationally feed the unlabeled video frames together with the labeled images into an image shadow detection network training. Specifically, we propose the Spatial and Temporal ICT, in which we define two new interpolation schemes, …
Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
ByteDance Intelligent Creation（字节跳动智能创作）；Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Jiao Tong University（上海交通大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Enhancing Multi-Camera People Tracking with Anchor-Guided Clustering and Spatio-Temporal Consistency ID Re-Assignment
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；National Center for High-Performance Computing（国家高性能计算中心）
提出一种结合锚定引导聚类与时空一致性的多摄像机多人员跟踪方法，在CVPR AI City Challenge 2023上以95.36% IDF1获第一名。
Learning Natural Consistency Representation for Face Forgery Video Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；RealAI
Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and recon…
Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Intelligent medical diagnosis has shown remarkable progress based on the large-scale datasets with precise annotations. However, fewer labeled images are available due to significantly expensive cost for annotating data by experts. To fully exploit the easily available unlabeled data, we propose a novel Spatio-Temporal Structure Consistent (STSC) learning framework. Specifically, a gram matrix is derived to combine the spatial structure consistency and temporal structure consistency together. This gram matrix captures the structural similarity among the representations of different training sa…
SOKENDAI (Graduate University for Advanced Studies)（综合研究大学院大学）；National Institute of Informatics（国立信息学研究所）
Detecting salient objects from a video requires exploiting both spatial and temporal knowledge included in the video. We propose a novel region-based multiscale spatiotemporal saliency detection method for videos, where static features and dynamic features computed from the low and middle levels are combined together. Our method utilizes such combined features spatially over each frame and, at the same time, temporally across frames using consistency between consecutive frames. Saliency cues in our method are analyzed through a multiscale segmentation model, and fused across scale levels, yiel…
Center for Research in Computer Vision（计算机视觉研究中心）；University of Central Florida（中佛罗里达大学）
In this work, we focus on semi-supervised learning for video action detection which utilizes both labeled as well as unlabeled data. We propose a simple end-to-end consistency based approach which effectively utilizes the unlabeled data. Video action detection requires both, action class prediction as well as a spatio-temporal localization of actions. Therefore, we investigate two types of constraints, classification consistency, and spatio-temporal consistency. The presence of predominant background and static regions in a video makes it challenging to utilize spatio-temporal consistency for …
Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated videos generally suffer from temporal and semantic inconsistency. To address this, many existing works adopt spatiotemporal consistency constraints incorporating temporal information based on motion estimation. However, the inaccuracies in the estimation of motion deteriorate the quality of the guidance towards spatiotemporal consistency, which leads to unstable translation. In th…
A spatio-temporal network for video semantic segmentation in surgical videos
Medtronic Digital Surgery（美敦力数字外科）；Wellcome/EPSRC Centre for Internventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科学中心）
Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurement
Systems Hub, the Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）系统中心）；Information Hub, the Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）信息中心）；School of Information Engineering, Sichuan Agricultural University（四川农业大学信息工程学院）；School of Computing and Information Technology, Great Bay University（大湾大学计算与信息学院）
We propose a new loss function that we call Laplacian loss, based on spatio-temporal Laplacian representation of the motion as a graph. This loss function is intended to be used in training models for motion reconstruction through 3D human pose estimation from videos. It compares the differential coordinates of the joints obtained from the graph representation of the ground truth against the one of the estimation. We design a fully convolutional temporal network for motion reconstruction to achieve better temporal consistency of estimation. We use this generic model to study the impact of our …
SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency
School of Computing and Information Technology, Great Bay University（大湾区大学 计算与信息技术学院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学 国家大数据系统计算技术工程实验室）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（新加坡科技研究局高性能计算研究所）
Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis
School of Electrical and Computer Engineering, College of Engineering, University of Tehran（德黑兰大学工程学院电气与计算机工程学院）；School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM)（基础科学研究所认知科学学院）
