Most current pipelines for spatio-temporal action localization connect frame-wise or clip-wise detection results to generate action proposals, where only local information is exploited and the efficiency is hindered by dense per-frame localization. In this paper, we propose Coarse-to-Fine Action Detector (CFAD),an original end-to-end trainable framework for efficient spatio-temporal action localization. The CFAD introduces a new paradigm that first estimates coarse spatio-temporal action tubes from video streams, and then refines the tubes' location based on key timestamps. This concept is implemented by two key components, the Coarse and Refine Modules in our framework. The parameterized modeling of long temporal information in the Coarse Module helps obtain accurate initial tube estimation, while the Refine Module selectively adjusts the tube location under the guidance of key timestamps. Against other methods, theproposed CFAD achieves competitive results on action detection benchmarks of UCF101-24, UCFSports and JHMDB-21 with inference speed that is 3.3x faster than the nearest competitors.
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Toyota Technological Institute（丰田工业大学）；RIKEN（理化学研究所）
The success of deep learning on video Action Recognition (AR) has motivated researchers to progressively promote related tasks from the coarse level to the fine-grained level. Compared with conventional AR which only predicts an action label for the entire video, Temporal Action Detection (TAD) has been investigated for estimating the start and end time for each action in videos. Taking TAD a step further, Spatiotemporal Action Detection (SAD) has been studied for localizing the action both spatially and temporally in videos. However, who performs the action, is generally ignored in SAD, while…
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Manitoba（曼尼托巴大学）
Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc. This work exploits an essential yet under-explored property of videos, the video continuity, to obtain supervision signals for self-supervised representation learning. Specifically, we formulate three novel continuity-related pretext tasks, i.e. continuity justification, discontinuity localization, and missing section approximation, that jointly supervise a shared backbone for video representation learning. This self-supervisi…
STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding
Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；Zhejiang University of Technology（浙江工业大学）；University of Chinese Academy of Sciences（中国科学院大学）
Tianjin University（天津大学）；Key Laboratory of Artificial Intelligence, Ministry of Education（教育部人工智能重点实验室）
As moving objects always draw more attention of human eyes, the temporal motive information is always exploited complementarily with spatial information to detect salient objects in videos. Although efficient tools such as optical flow have been proposed to extract temporal motive information, it often encounters difficulties when used for saliency detection due to the movement of camera or the partial movement of salient objects. In this paper, we investigate the complimentary roles of spatial and temporal information and propose a novel dynamic spatiotemporal network (DS-Net) for more effect…
Queen Mary University of London（伦敦玛丽女王大学）；Onfido Research London（Onfido伦敦研究院）
WiFi-based sensing for human activity recognition (HAR) has recently become a hot topic as it brings great benefits when compared with video-based HAR, such as eliminating the demands of line-of-sight (LOS) and preserving privacy. Making the WiFi signals to 'see' the action, however, is quite coarse and thus still in its infancy. An end-to-end spatiotemporal WiFi signal neural network (STWNN) is proposed to enable WiFi-only sensing in both line-of-sight and non-line-of-sight scenarios. Especially, the 3D convolution module is able to explore the spatiotemporal continuity of WiFi signals, and t…
School of Computer Science and Technology, UCAS（中国科学院大学计算机科学与技术学院）；JD Digits（京东数科）；Key Laboratory of Big Data Mining and Knowledge Management, CAS（中国科学院大数据挖掘与知识管理重点实验室）；Key Laboratory of Intell. Info. Process. (IIP), Inst. of Computi. Tech., CAS（中国科学院计算技术研究所智能信息处理重点实验室）
In this paper, we present a unified, end-to-end trainable spatiotemporal CNN model for VOS, which consists of two branches, i.e., the temporal coherence branch and the spatial segmentation branch. Specifically, the temporal coherence branch pretrained in an adversarial fashion from unlabeled video data, is designed to capture the dynamic appearance and motion cues of video sequences to guide object segmentation. The spatial segmentation branch focuses on segmenting objects accurately based on the learned appearance and motion cues. To obtain accurate segmentation results, we design a coarse-to…
Spatiotemporal Satellite Image Downscaling with Transfer Encoders and Autoregressive Generative Models
University of Toronto（多伦多大学）；Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院）；Harvard Medical School（哈佛医学院）；VA Healthcare System Boston, U.S. Department of Veterans Affairs（美国退伍军人事务部波士顿医疗系统）
Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology（科技大学）；Anhui Engineering Laboratory for Intelligent Driving Technology and Application（安徽省智能驾驶技术与应用工程实验室）；Innovation Research Institute of Robotics and Intelligent Manufacturing Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）
Ground segmentation is an important preprocessing task for autonomous vehicles (AVs) with 3D LiDARs. To solve the problem of existing ground segmentation methods being very difficult to balance accuracy and computational complexity, a fast point cloud ground segmentation approach based on a coarse-to-fine Markov random field (MRF) method is proposed. The method uses an improved elevation map for ground coarse segmentation, and then uses spatiotemporal adjacent points to optimize the segmentation results. The processed point cloud is classified into high-confidence obstacle points, ground point…
Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition
Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院）
Four-dimensional magnetic resonance imaging (4D-MRI) is an emerging technique for tumor motion management in image-guided radiation therapy (IGRT). However, current 4D-MRI suffers from low spatial resolution and strong motion artifacts owing to the long acquisition time and patients' respiratory variations; these limitations, if not managed properly, can adversely affect treatment planning and delivery in IGRT. Herein, we developed a novel deep learning framework called the coarse-super-resolution-fine network (CoSF-Net) to achieve simultaneous motion estimation and super-resolution in a unifi…
There is significant progress in recognizing traditional human activities from videos focusing on highly distinctive actions involving discriminative body movements, body-object and/or human-human interactions. Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. To address this, we propose a novel framework by exploiting the spatiotemporal attention to model the subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable g…
Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer
Department of Computational Engineering, Lappeenranta-Lahti University of Technology LUT, Finland（拉彭兰塔-拉赫蒂理工大学计算工程系，芬兰）；Faculty of Information Technology, Brno University of Technology, Czech Republic（布尔诺理工大学信息技术学院，捷克共和国）
Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学PCA实验室）；PCA Lab, School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学PCA实验室）
