We propose MASTAF, a Model-Agnostic Spatio-Temporal Attention Fusion network for few-shot video classification. MASTAF takes input from a general video spatial and temporal representation,e.g., using 2D CNN, 3D CNN, and Video Transformer. Then, to make the most of such representations, we use self- and cross-attention models to highlight the critical spatio-temporal region to increase the inter-class variations and decrease the intra-class variations. Last, MASTAF applies a lightweight fusion network and a nearest neighbor classifier to classify each query video. We demonstrate that MASTAF improves the state-of-the-art performance on three few-shot video classification benchmarks(UCF101, HMDB51, and Something-Something-V2), e.g., by up to 91.6%, 69.5%, and 60.7% for five-way one-shot video classification, respectively.
Background subtraction (BGS) is a fundamental video processing task which is a key component of many applications. Deep learning-based supervised algorithms achieve very good perforamnce in BGS, however, most of these algorithms are optimized for either a specific video or a group of videos, and their performance decreases dramatically when applied to unseen videos. Recently, several papers addressed this problem and proposed video-agnostic supervised BGS algorithms. However, nearly all of the data augmentations used in these algorithms are limited to the spatial domain and do not account for …
STGV: Spatio-Temporal Hash Encoding for Gaussian-based Video Representation
College of Electronics and Information Engineering（电子与信息工程学院）；College of Applied Technology（应用技术学院）；School of Artificial Intelligence（人工智能学院）；Research Center of Networks and Communications（网络与通信研究中心）；Pengcheng Laboratory（鹏城实验室）
UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines
MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiaotong University（上海交通大学）；Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong（香港中文大学信兴高等工程研究所）
University of California, Santa Barbara（加州大学圣巴巴拉分校）
3D human motion prediction is a research area of high significance and a challenge in computer vision. It is useful for the design of many applications including robotics and autonomous driving. Traditionally, autogregressive models have been used to predict human motion. However, these models have high computation needs and error accumulation that make it difficult to use them for realtime applications. In this paper, we present a non-autogressive model for human motion prediction. We focus on learning spatio-temporal representations non-autoregressively for generation of plausible future mot…
Huazhong University of Science and Technology（华中科技大学）；Columbia University（哥伦比亚大学）；Dalian University of Technology（大连理工大学）
This paper addresses the task of segmenting class-agnostic objects in semi-supervised setting. Although previous detection based methods achieve relatively good performance, these approaches extract the best proposal by a greedy strategy, which may lose the local patch details outside the chosen candidate. In this paper, we propose a novel spatiotemporal graph neural network (STG-Net) to reconstruct more accurate masks for video object segmentation, which captures the local contexts by utilizing all proposals. In the spatial graph, we treat object proposals of a frame as nodes and represent th…
EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection
N2TP Technology Solution JSC（N2TP技术解决方案有限公司）；College of Technology, National Economics University（国家经济大学技术学院）；University of Science, Vietnam National University（越南国家大学科学大学）；Faculty of AI and Data Science, Phenikaa University（Phenikaa大学人工智能与数据科学学院）；The Saigon International University（西贡国际大学）；College of Informatics, Yuan Ze University（元智大学资讯学院）
DuMeta++: Spatiotemporal Dual Meta-Learning for Generalizable Few-Shot Brain Tissue Segmentation Across Diverse Ages
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Key Laboratory of Biomedical Information Engineering of Ministry of Education, School of Life Science and Technology, Xi’an Jiaotong University（教育部生物医学信息工程重点实验室，西安交通大学生命科学与技术学院）
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学协同媒体网络创新中心）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）；Shanghai Jiao Tong University（上海交通大学）
We propose a multiscale spatio-temporal graph neural network (MST-GNN) to predict the future 3D skeleton-based human poses in an action-category-agnostic manner. The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales. Different from many previous hierarchical structures, our multiscale spatio-temporal graph is built in a data-adaptive fashion, which captures nonphysical, yet motion-based relations. The key module of MST-GNN is a multiscale spatio-temporal graph computational unit (MST-GCU) based on the tr…
VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation
College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（科学、技术与研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（科学、技术与研究局前沿人工智能研究中心）
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）；Dr. Ing. h.c. F. Porsche AG（德意志联邦汽车工业联合会）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
Dalian University of Technology（大连理工大学）；Tianjin University（天津大学）
For video recognition task, a global representation summarizing the whole contents of the video snippets plays an important role for the final performance. However, existing video architectures usually generate it by using a simple, global average pooling (GAP) method, which has limited ability to capture complex dynamics of videos. For image recognition task, there exist evidences showing that covariance pooling has stronger representation ability than GAP. Unfortunately, such plain covariance pooling used in image recognition is an orderless representative, which cannot model spatio-temporal…
This thesis focuses on video understanding for human action and interaction recognition. We start by identifying the main challenges related to action recognition from videos and review how they have been addressed by current methods. Based on these challenges, and by focusing on the temporal aspect of actions, we argue that current fixed-sized spatio-temporal kernels in 3D convolutional neural networks (CNNs) can be improved to better deal with temporal variations in the input. Our contributions are based on the enlargement of the convolutional receptive fields through the introduction of spa…
DeepMind（深度思维）；University of Oxford（牛津大学）；Carnegie Mellon University（卡内基梅隆大学）
We introduce the Action Transformer model for recognizing and localizing human actions in video clips. We repurpose a Transformer-style architecture to aggregate features from the spatiotemporal context around the person whose actions we are trying to classify. We show that by using high-resolution, person-specific, class-agnostic queries, the model spontaneously learns to track individual people and to pick up on semantic context from the actions of others. Additionally its attention mechanism learns to emphasize hands and faces, which are often crucial to discriminate an action - all without…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Universidad de los Andes（安第斯大学）；University of Freiburg（弗莱堡大学）
Instance-level video segmentation requires a solid integration of spatial and temporal information. However, current methods rely mostly on domain-specific information (online learning) to produce accurate instance-level segmentations. We propose a novel approach that relies exclusively on the integration of generic spatio-temporal attention cues. Our strategy, named Multi-Attention Instance Network (MAIN), overcomes challenging segmentation scenarios over arbitrary videos without modelling sequence- or instance-specific knowledge. We design MAIN to segment multiple instances in a single forwa…
The objective of this work is to learn an object-centric video representation, with the aim of improving transferability to novel tasks, i.e., tasks different from the pre-training task of action classification. To this end, we introduce a new object-centric video recognition model based on a transformer architecture. The model learns a set of object-centric summary vectors for the video, and uses these vectors to fuse the visual and spatio-temporal trajectory 'modalities' of the video clip. We also introduce a novel trajectory contrast loss to further enhance objectness in these summary vecto…
ETH Zurich（苏黎世联邦理工学院）；Beijing Institute of Technology（北京理工大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
This paper addresses the task of unsupervised video multi-object segmentation. Current approaches follow a two-stage paradigm: 1) detect object proposals using pre-trained Mask R-CNN, and 2) conduct generic feature matching for temporal association using re-identification techniques. However, the generic features, widely used in both stages, are not reliable for characterizing unseen objects, leading to poor generalization. To address this, we introduce a novel approach for more accurate and efficient spatio-temporal segmentation. In particular, to address \textbf{instance discrimination}, we …
