In this paper, we propose a spatio-temporal contextual network, STC-Flow, for optical flow estimation. Unlike previous optical flow estimation approaches with local pyramid feature extraction and multi-level correlation, we propose a contextual relation exploration architecture by capturing rich long-range dependencies in spatial and temporal dimensions. Specifically, STC-Flow contains three key context modules - pyramidal spatial context module, temporal context correlation module and recurrent residual contextual upsampling module, to build the relationship in each stage of feature extraction, correlation, and flow reconstruction, respectively. Experimental results indicate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on the Sintel dataset and the KITTI 2012/2015 datasets.
STCMOT: Spatio-Temporal Cohesion Learning for UAV-Based Multiple Object Tracking
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Optics and Electronics（光电技术研究所）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Department of Computer Science and Technology, China University of Mining & Technology, Beijing（中国矿业大学（北京）计算机科学与技术系）；School of Computing & Communications, Lancaster University（兰卡斯特大学计算与通信学院）；University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）；Noah’s Ark Lab, Huawei Technologies Co. Ltd.（华为技术有限公司诺亚方舟实验室）；Department of Electrical and Computer Engineering, University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校电气与计算机工程系）
Skeleton-based action recognition task is entangled with complex spatio-temporal variations of skeleton joints, and remains challenging for Recurrent Neural Networks (RNNs). In this work, we propose a temporal-then-spatial recalibration scheme to alleviate such complex variations, resulting in an end-to-end Memory Attention Networks (MANs) which consist of a Temporal Attention Recalibration Module (TARM) and a Spatio-Temporal Convolution Module (STCM). Specifically, the TARM is deployed in a residual learning module that employs a novel attention learning network to recalibrate the temporal at…
Industrial smoke emissions present a serious threat to natural ecosystems and human health. Prior works have shown that using computer vision techniques to identify smoke is a low cost and convenient method. However, industrial smoke detection is a challenging task because industrial emission particles are often decay rapidly outside the stacks or facilities and steam is very similar to smoke. To overcome these problems, a novel Spatio-Temporal Cross Network (STCNet) is proposed to recognize industrial smoke emissions. The proposed STCNet involves a spatial pathway to extract texture features …
Tencent Youtu Lab（腾讯优图实验室）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）
Video Instance Segmentation (VIS) is a task that simultaneously requires classification, segmentation, and instance association in a video. Recent VIS approaches rely on sophisticated pipelines to achieve this goal, including RoI-related operations or 3D convolutions. In contrast, we present a simple and efficient single-stage VIS framework based on the instance segmentation method CondInst by adding an extra tracking head. To improve instance association accuracy, a novel bi-directional spatio-temporal contrastive learning strategy for tracking embedding across frames is proposed. Moreover, a…
Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）信息通信研究院（I2R））；School of Computing, National University of Singapore（新加坡国立大学计算学院）；SCSE, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Laboratory of Image Sciences and Technology, Southeast University（东南大学影像科学与技术实验室）；Ginkgo LLC（银杏有限责任公司）
This paper presents a novel mid-level representation for action recognition, named spatio-temporal aware non-negative component representation (STANNCR). The proposed STANNCR is based on action component and incorporates the spatial-temporal information. We first introduce a spatial-temporal distribution vector (STDV) to model the distributions of local feature locations in a compact and discriminative manner. Then we employ non-negative matrix factorization (NMF) to learn the action components and encode the video samples. The action component considers the correlations of visual words, which…
Spatio-Temporal video grounding (STVG) focuses on retrieving the spatio-temporal tube of a specific object depicted by a free-form textual expression. Existing approaches mainly treat this complicated task as a parallel frame-grounding problem and thus suffer from two types of inconsistency drawbacks: feature alignment inconsistency and prediction inconsistency. In this paper, we present an end-to-end one-stage framework, termed Spatio-Temporal Consistency-Aware Transformer (STCAT), to alleviate these issues. Specially, we introduce a novel multi-modal template as the global objective to addre…
V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shenzhen Loop Area Institute（深圳河套学院）；Sichuan University（四川大学）；EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）
Westlake Institute for Advanced Study（西湖高等研究院）；Fudan University（复旦大学）；Meta AI；Westlake University（西湖大学）；Meituan（美团）；Shandong First Medical University & Shandong Academy of Medical Sciences（山东第一医科大学（山东省医学科学院））
Instance segmentation in videos, which aims to segment and track multiple objects in video frames, has garnered a flurry of research attention in recent years. In this paper, we present a novel weakly supervised framework with \textbf{S}patio-\textbf{T}emporal \textbf{C}ollaboration for instance \textbf{Seg}mentation in videos, namely \textbf{STC-Seg}. Concretely, STC-Seg demonstrates four contributions. First, we leverage the complementary representations from unsupervised depth estimation and optical flow to produce effective pseudo-labels for training deep networks and predicting high-quali…
ESAT-PSI；KU Leuven（鲁汶大学）；CV:HCI；KIT（卡尔斯鲁厄理工学院）；Karlsruhe, University of Bonn（波恩大学）；Sensifai
The work in this paper is driven by the question if spatio-temporal correlations are enough for 3D convolutional neural networks (CNN)? Most of the traditional 3D networks use local spatio-temporal features. We introduce a new block that models correlations between channels of a 3D CNN with respect to temporal and spatial features. This new block can be added as a residual unit to different parts of 3D CNNs. We name our novel block 'Spatio-Temporal Channel Correlation' (STC). By embedding this block to the current state-of-the-art architectures such as ResNext and ResNet, we improved the perfo…
A Spatio-temporal Continuous Network for Stochastic 3D Human Motion Prediction
Colleage of Computer Science and Technology, Dalian University of Technology（计算机科学与技术学院，大连理工大学）；Centre for Frontier AI Research, Institute of High Performance Computing, A*STAR（前沿人工智能研究中心，高性能计算研究所）；School of Software Engineering, Dalian University（软件工程学院，大连大学）
STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家实验室）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人工智能与机器人学院，西安交通大学）
Hierarchical Spatio-temporal Segmentation Network for Ejection Fraction Estimation in Echocardiography Videos
PCA Lab（PCA实验室）；Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education（教育部高维信息智能感知与系统重点实验室）；School of Computer Science and Engineering（计算机科学与工程学院）；Nanjing University of Science and Technology（南京理工大学）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Information Technology R&D Innovation Center of Peking University（北京大学信息技术创新研发中心）
Video prediction aims to predict future frames by modeling the complex spatiotemporal dynamics in videos. However, most of the existing methods only model the temporal information and the spatial information for videos in an independent manner but haven't fully explored the correlations between both terms. In this paper, we propose a SpatioTemporal-Aware Unit (STAU) for video prediction and beyond by exploring the significant spatiotemporal correlations in videos. On the one hand, the motion-aware attention weights are learned from the spatial states to help aggregate the temporal states in th…
Learning from Single Timestamps: Complexity Estimation in Laparoscopic Cholecystectomy
UCL Hawkes Institute, University College London（UCL哈维斯研究所，伦敦大学学院）；Medtronic plc.（美敦力公司）；Dept of Medical Physics & Biomedical Engineering, University College London（医学物理与生物医学工程系，伦敦大学学院）；Dept of Computer Science, University College London（计算机科学系，伦敦大学学院）
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Alibaba Group（阿里巴巴集团）
As a fundamental and challenging problem in computer vision, hand pose estimation aims to estimate the hand joint locations from depth images. Typically, the problem is modeled as learning a mapping function from images to hand joint coordinates in a data-driven manner. In this paper, we propose Context-Aware Deep Spatio-Temporal Network (CADSTN), a novel method to jointly model the spatio-temporal properties for hand pose estimation. Our proposed network is able to learn the representations of the spatial information and the temporal structure from the image sequences. Moreover, by adopting a…
Pedestrian trajectory prediction is essential for collision avoidance in autonomous driving and robot navigation. However, predicting a pedestrian's trajectory in crowded environments is non-trivial as it is influenced by other pedestrians' motion and static structures that are present in the scene. Such human-human and human-space interactions lead to non-linearities in the trajectories. In this paper, we present a new spatio-temporal graph based Long Short-Term Memory (LSTM) network for predicting pedestrian trajectory in crowded environments, which takes into account the interaction with st…
University of Chinese Academy of Sciences（中国科学院大学）；Meitu（美图公司）；National Computer network Emergency Response technical Team/Coordination Center of China（国家计算机网络应急技术处理协调中心）
Local features at neighboring spatial positions in feature maps have high correlation since their receptive fields are often overlapped. Self-attention usually uses the weighted sum (or other functions) with internal elements of each local feature to obtain its weight score, which ignores interactions among local features. To address this, we propose an effective interaction-aware self-attention model inspired by PCA to learn attention maps. Furthermore, since different layers in a deep network capture feature maps of different scales, we use these feature maps to construct a spatial pyramid a…
Vulnerability-Aware Spatio-Temporal Learning for Generalizable Deepfake Video Detection
CVI 2 , SnT, University of Luxembourg（CVI 2 ，SnT，卢森堡大学）；Cristal Laboratory, National School of Computer Sciences, University of Manouba（Cristal实验室，国家计算机科学学校，曼努巴大学）
Most existing transformer based video instance segmentation methods extract per frame features independently, hence it is challenging to solve the appearance deformation problem. In this paper, we observe the temporal information is important as well and we propose TAFormer to aggregate spatio-temporal features both in transformer encoder and decoder. Specifically, in transformer encoder, we propose a novel spatio-temporal joint multi-scale deformable attention module which dynamically integrates the spatial and temporal information to obtain enriched spatio-temporal features. In transformer d…
4D-GSW: Kinematic-Aware Spatio-Temporal Consistent Watermarking for 4D Gaussian Splatting
Southeast University（东南大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy（广东人工智能与数字经济实验室）；University of Chinese Academy of Sciences（中国科学院大学）
College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；Huawei Cloud & AI（华为云与人工智能）
Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of a queried object according to the given sentence. Currently, most existing grounding methods are restricted to well-aligned segment-sentence pairs. In this paper, we explore spatio-temporal video grounding on unaligned data and multi-form sentences. This challenging task requires to capture critical object relations to identify the queried target. However, existing approaches cannot distinguish notable objects and remain in ineffective relation modeling between unnecessary objects. Thus, we propose a novel object-awar…
Huazhong University of Science and Technology（华中科技大学）；Megvii Inc. (Face++)（旷视科技（Face++））
Current state-of-the-art approaches for spatio-temporal action detection have achieved impressive results but remain unsatisfactory for temporal extent detection. The main reason comes from that, there are some ambiguous states similar to the real actions which may be treated as target actions even by a well-trained network. In this paper, we define these ambiguous samples as "transitional states", and propose a Transition-Aware Context Network (TACNet) to distinguish transitional states. The proposed TACNet includes two main components, i.e., temporal context detector and transition-aware cla…
University of Cambridge（剑桥大学）；Middle East Technical University（中东技术大学）
Most state-of-the-art approaches for Facial Action Unit (AU) detection rely upon evaluating facial expressions from static frames, encoding a snapshot of heightened facial activity. In real-world interactions, however, facial expressions are usually more subtle and evolve in a temporal manner requiring AU detection models to learn spatial as well as temporal information. In this paper, we focus on both spatial and spatio-temporal features encoding the temporal evolution of facial AU activation. For this purpose, we propose the Action Unit Lifecycle-Aware Capsule Network (AULA-Caps) that perfor…
Context-aware Sparse Spatiotemporal Learning for Event-based Vision
Faculty of Aerospace Engineering, Delft University of Technology（代尔夫特理工大学航空航天工程学院）；Department of Advanced Computing Sciences, Maastricht University（马斯特里赫特大学先进计算科学系）
Spatiotemporal Learning with Context-aware Video Tubelets for Ultrasound Video Analysis
Philips（飞利浦）；Columbia University Vagelos College of Physicians and Surgeons（哥伦比亚大学瓦格洛斯内科医生与外科医生学院）；MedStar Washington Hospital Center（MedStar华盛顿医院中心）；Yale University School of Medicine（耶鲁大学医学院）
Georgia Institute of Technology（佐治亚理工学院）；SRI International（SRI国际）
This paper presents a novel approach for the Vision-and-Language Navigation (VLN) task in continuous 3D environments, which requires an autonomous agent to follow natural language instructions in unseen environments. Existing end-to-end learning-based VLN methods struggle at this task as they focus mostly on utilizing raw visual observations and lack the semantic spatio-temporal reasoning capabilities which is crucial in generalizing to new environments. In this regard, we present a hybrid transformer-recurrence model which focuses on combining classical semantic mapping techniques with a lear…
