In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.
Singapore University of Technology and Design（新加坡科技设计大学）；Monash University（莫纳什大学）；Lancaster University（兰卡斯特大学）；ByteDance（字节跳动）；Harbin Institute of Technology（哈尔滨工业大学）
The goal of fine-grained action recognition is to successfully discriminate between action categories with subtle differences. To tackle this, we derive inspiration from the human visual system which contains specialized regions in the brain that are dedicated towards handling specific tasks. We design a novel Dynamic Spatio-Temporal Specialization (DSTS) module, which consists of specialized neurons that are only activated for a subset of samples that are highly similar. During training, the loss forces the specialized neurons to learn discriminative fine-grained differences to distinguish be…
Spatiotemporal predictive learning (ST-PL) aims at predicting the subsequent frames via limited observed sequences, and it has broad applications in the real world. However, learning representative spatiotemporal features for prediction is challenging. Moreover, chaotic uncertainty among consecutive frames exacerbates the difficulty in long-term prediction. This paper concentrates on improving prediction quality by enhancing the correspondence between the previous context and the current state. We carefully design Detail Context Block (DCB) to extract fine-grained details and improve the isola…
Joint segmentation and classification of fine-grained actions is important for applications of human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large-scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. We propose a model for action segmentation which combines low-level spatiotemporal features with a high-level segmental classifier. Our spatiotemporal CNN is comprised of a spatial component that uses convolutional filters to capture information about objec…
Decoupling Spatio-Temporal Adapter for Fine-Grained Badminton Action Localization
School of Economics and Management, Beihang University（北京航空航天大学经济管理学院）；Key Laboratory of Data Intelligence and Management, Beihang University, Ministry of Industry and Information Technology（信息产业部北京航空航天大学数据智能与管理重点实验室）
Lip-reading aims to recognize speech content from videos via visual analysis of speakers' lip movements. This is a challenging task due to the existence of homophemes-words which involve identical or highly similar lip movements, as well as diverse lip appearances and motion patterns among the speakers. To address these challenges, we propose a novel lip-reading model which captures not only the nuance between words but also styles of different speakers, by a multi-grained spatio-temporal modeling of the speaking process. Specifically, we first extract both frame-level fine-grained features an…
Fine-grained Context and Multi-modal Alignment for Freehand 3D Ultrasound Reconstruction
Shenzhen University（深圳大学）；School of Biomedical Engineering, Shenzhen University Medical School（深圳大学医学院生物医学工程学院）；National-Regional Key Technology Engineering Laboratory for Medical Ultrasound（国家区域医学超声关键技术工程实验室）；Marshall Laboratory of Biomedical Engineering, Shenzhen University（深圳大学马歇尔生物医学工程实验室）
MILA, Montreal Institute for Learning Algorithms（MILA蒙特利尔学习算法研究所）；Paris-Saclay University（巴黎萨克雷大学）；CentraleSupélec Paris (ECP)（巴黎中央理工学院）；INRIA Paris（法国国家信息与自动化研究所巴黎分部）；Operations Research Center, MIT（麻省理工学院运筹学中心）；Université de Montréal（蒙特利尔大学）；University of Toronto（多伦多大学）；McGill University（麦吉尔大学）
Fine-grained classification aims at distinguishing between items with similar global perception and patterns, but that differ by minute details. Our primary challenges come from both small inter-class variations and large intra-class variations. In this article, we propose to combine several innovations to improve fine-grained classification within the use-case of wildlife, which is of practical interest for experts. We utilize geo-spatiotemporal data to enrich the picture information and further improve the performance. We also investigate state-of-the-art methods for handling the imbalanced …
FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment
School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Fine-Grained Traffic Inference from Road to Lane via Spatio-Temporal Graph Node Generation
Fudan University（复旦大学）；Zhuhai Fudan Innovation Research Institute（珠海复旦创新研究院）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Guangzhou University（广州大学）；Shanghai Jiao Tong University（上海交通大学）
Modern self-supervised learning algorithms typically enforce persistency of instance representations across views. While being very effective on learning holistic image and video representations, such an objective becomes sub-optimal for learning spatio-temporally fine-grained features in videos, where scenes and instances evolve through space and time. In this paper, we present Contextualized Spatio-Temporal Contrastive Learning (ConST-CL) to effectively learn spatio-temporally fine-grained video representations via self-supervision. We first design a region-based pretext task which requires …
Gait recognition is an important recognition technology, because gait is not easy to camouflage and does not need cooperation to recognize subjects. However, many existing methods are inadequate in preserving both temporal information and fine-grained information, thus reducing its discrimination. This problem is more serious when the subjects with similar walking postures are identified. In this paper, we try to enhance the discrimination of spatio-temporal gait features from two aspects: effective extraction of spatio-temporal gait features and reasonable refinement of extracted features. Th…
STeP-Diff: Spatio-Temporal Physics-Informed Diffusion Models for Mobile Fine-Grained Pollution Forecasting
Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）；Department of Electronic Engineering, Beijing National Research Center for Information Science and Tech- nology (BNRist), Tsinghua University, China（电子工程系，北京信息科学与技术国家研究中心（BNRist），清华大学）；Shenzhen Smartcity Communication, Shenzhen, China（深圳智慧城市通信）；College of Soil and Water Conservation, Southwest Forestry University, China（水土保持学院，西南林业大学）；College of Civil Engineering, Southwest Forestry University, China（土木工程学院，西南林业大学）
University of Bordeaux（波尔多大学）；CNRS（法国国家科学研究中心）；Bordeaux INP（波尔多国立高等物理、数学与信息学院）；University of La Rochelle（拉罗谢尔大学）
The paper addresses the problem of recognition of actions in video with low inter-class variability such as Table Tennis strokes. Two stream, "twin" convolutional neural networks are used with 3D convolutions both on RGB data and optical flow. Actions are recognized by classification of temporal windows. We introduce 3D attention modules and examine their impact on classification efficiency. In the context of the study of sportsmen performances, a corpus of the particular actions of table tennis strokes is considered. The use of attention blocks in the network speeds up the training step and i…
FireSentry: A Multi-Modal Spatio-temporal Benchmark Dataset for Fine-Grained Wildfire Spread Forecasting
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；College of Soil and Water Conservation, Southwest Forestry University（西南林业大学水土保持学院）；College of Civil Engineering, Southwest Forestry University（西南林业大学土木工程学院）
Appearance information alone is often not sufficient to accurately differentiate between fine-grained visual categories. Human experts make use of additional cues such as where, and when, a given image was taken in order to inform their final decision. This contextual information is readily available in many online image collections but has been underutilized by existing image classifiers that focus solely on making predictions based on the image contents. We propose an efficient spatio-temporal prior, that when conditioned on a geographical location and time, estimates the probability that a …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research AI（IBM 研究院人工智能部门）
Fine-grained action detection is an important task with numerous applications in robotics and human-computer interaction. Existing methods typically utilize a two-stage approach including extraction of local spatio-temporal features followed by temporal modeling to capture long-term dependencies. While most recent papers have focused on the latter (long-temporal modeling), here, we focus on producing features capable of modeling fine-grained motion more efficiently. We propose a novel locally-consistent deformable convolution, which utilizes the change in receptive fields and enforces a local …
EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；The University of Tokyo（东京大学）；Fudan University（复旦大学）；Nanjing University（南京大学）
The task of skeleton-based action recognition remains a core challenge in human-centred scene understanding due to the multiple granularities and large variation in human motion. Existing approaches typically employ a single neural representation for different motion patterns, which has difficulty in capturing fine-grained action classes given limited training data. To address the aforementioned problems, we propose a novel multi-granular spatio-temporal graph network for skeleton-based action classification that jointly models the coarse- and fine-grained skeleton motion patterns. To this end…
Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition
University of Chinese Academy of Sciences（中国科学院大学）；National Key Laboratory of Space Integrated Information System（空间信息集成系统国家重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Technology（计算机科学与技术系）
STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding
Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；Zhejiang University of Technology（浙江工业大学）；University of Chinese Academy of Sciences（中国科学院大学）
The University of Hong Kong（香港大学）；Tencent AI Lab（腾讯人工智能实验室）
In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatio-temporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their ma…
T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Video
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）
Vehicle re-identification (re-ID) aims to discover and match the target vehicles from a gallery image set taken by different cameras on a wide range of road networks. It is crucial for lots of applications such as security surveillance and traffic management. The remarkably similar appearances of distinct vehicles and the significant changes of viewpoints and illumination conditions take grand challenges to vehicle re-ID. Conventional solutions focus on designing global visual appearances without sufficient consideration of vehicles' spatiotamporal relationships in different images. In this pa…
Spatio-temporal representational learning has been widely adopted in various fields such as action recognition, video object segmentation, and action anticipation. Previous spatio-temporal representational learning approaches primarily employ ConvNets or sequential models,e.g., LSTM, to learn the intra-frame and inter-frame features. Recently, Transformer models have successfully dominated the study of natural language processing (NLP), image classification, etc. However, the pure-Transformer based spatio-temporal learning can be prohibitively costly on memory and computation to extract fine-g…
Video Anomaly Detection (VAD) is an important topic in computer vision. Motivated by the recent advances in self-supervised learning, this paper addresses VAD by solving an intuitive yet challenging pretext task, i.e., spatio-temporal jigsaw puzzles, which is cast as a multi-label fine-grained classification problem. Our method exhibits several advantages over existing works: 1) the spatio-temporal jigsaw puzzles are decoupled in terms of spatial and temporal dimensions, responsible for capturing highly discriminative appearance and motion features, respectively; 2) full permutations are used …
Huazhong University of Science and Technology（华中科技大学）；Peking University（北京大学）；Wangxuan Institute of Computer Technology（王选计算机研究所）
Temporal sentence grounding (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed video.Most existing methods extract frame-grained features or object-grained features by 3D ConvNet or detection network under a conventional TSG framework, failing to capture the subtle differences between frames or to model the spatio-temporal behavior of core persons/objects. In this paper, we introduce a new perspective to address the TSG task by tracking pivotal objects and activities to learn more fine-grained spatio-temporal behaviors. Speci…
STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models
Ho Chi Minh University of Technology（胡志明理工大学）；Vietnamese-German University（越德大学）；Ho Chi Minh University of Science（胡志明理工大学）；University of Information Technology（信息科技大学）；Vietnam National University, Ho Chi Minh city（越南国家大学，胡志明市）
Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Queensland University of Technology（昆士兰科技大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；University of Queensland（昆士兰大学）；University of Adelaide（阿德莱德大学）
Fine-grained classification is a relatively new field that has concentrated on using information from a single image, while ignoring the enormous potential of using video data to improve classification. In this work we present the novel task of video-based fine-grained object classification, propose a corresponding new video dataset, and perform a systematic study of several recent deep convolutional neural network (DCNN) based approaches, which we specifically adapt to the task. We evaluate three-dimensional DCNNs, two-stream DCNNs, and bilinear DCNNs. Two forms of the two-stream approach are…
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代国家重点软件实验室）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Shanghai AI Lab（上海人工智能实验室）
