Video classification researches that have recently attracted attention are the fields of temporal modeling and 3D efficient architecture. However, the temporal modeling methods are not efficient or the 3D efficient architecture is less interested in temporal modeling. For bridging the gap between them, we propose an efficient temporal modeling 3D architecture, called VoV3D, that consists of a temporal one-shot aggregation (T-OSA) module and depthwise factorized component, D(2+1)D. The T-OSA is devised to build a feature hierarchy by aggregating temporal features with different temporal receptive fields. Stacking this T-OSA enables the network itself to model short-range as well as long-range temporal relationships across frames without any external modules. Inspired by kernel factorization and channel factorization, we also design a depthwise spatiotemporal factorization module, named, D(2+1)D that decomposes a 3D depthwise convolution into two spatial and temporal depthwise convolutions for making our network more lightweight and efficient. By using the proposed temporal modeling method (T-OSA), and the efficient factorized component (D(2+1)D), we construct two types of VoV3D networks, VoV3D-M and VoV3D-L. Thanks to its efficiency and effectiveness of temporal modeling, VoV3D-L has 6x fewer model parameters and 16x less computation, surpassing a state-of-the-art temporal modeling method on both Something-Something and Kinetics-400. Furthermore, VoV3D shows better temporal modeling ability than a state-of-the-art efficient 3D architecture, X3D having comparable model capacity. We hope that VoV3D can serve as a baseline for efficient video classification.
MIT-IBM Waston AI Lab（MIT-IBM沃森人工智能实验室）；IBM T.J. Waston Research Center（IBM托马斯·J·沃森研究中心）
Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets. This results in a need for large GPU clusters to train and evaluate such architectures. To address this problem, we present a lightweight and memory-friendly architecture for action recognition that performs on par with or better than current architectures by using only a fraction of resources. The proposed architecture is based on a combination of a deep subnet operating on low-resolution frames with a compact subnet operating on high-resolution frames, allowing for high efficiency and accu…
Traditional weather forecasting relies on domain expertise and computationally intensive numerical simulation systems. Recently, with the development of a data-driven approach, weather forecasting based on deep learning has been receiving attention. Deep learning-based weather forecasting has made stunning progress, from various backbone studies using CNN, RNN, and Transformer to training strategies using weather observations datasets with auxiliary inputs. All of this progress has contributed to the field of weather forecasting; however, many elements and complex structures of deep learning m…
Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer
Shanghai Jiao Tong University（上海交通大学）；Xinhua Hospital Affiliated to Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属新华医院）；Rice University（莱斯大学）；University of Münster（明斯特大学）
Tianjin University（天津大学）；Key Laboratory of Artificial Intelligence, Ministry of Education（教育部人工智能重点实验室）
As moving objects always draw more attention of human eyes, the temporal motive information is always exploited complementarily with spatial information to detect salient objects in videos. Although efficient tools such as optical flow have been proposed to extract temporal motive information, it often encounters difficulties when used for saliency detection due to the movement of camera or the partial movement of salient objects. In this paper, we investigate the complimentary roles of spatial and temporal information and propose a novel dynamic spatiotemporal network (DS-Net) for more effect…
Beijing Institute of Technology（北京理工大学）；Baidu Research（百度研究院）；Inception Institute of Artificial Intelligence（人工智能起源研究所）；National Engineering Laboratory of Deep Learning Technology and Application（深度学习技术及应用国家工程实验室）；University of Kentucky（肯塔基大学）
Existing LiDAR-based 3D object detectors usually focus on the single-frame detection, while ignoring the spatiotemporal information in consecutive point cloud frames. In this paper, we propose an end-to-end online 3D video object detector that operates on point cloud sequences. The proposed model comprises a spatial feature encoding component and a spatiotemporal feature aggregation component. In the former component, a novel Pillar Message Passing Network (PMPNet) is proposed to encode each discrete point cloud frame. It adaptively collects information for a pillar node from its neighbors by …
Current methods for spatiotemporal action tube detection often extend a bounding box proposal at a given keyframe into a 3D temporal cuboid and pool features from nearby frames. However, such pooling fails to accumulate meaningful spatiotemporal features if the position or shape of the actor shows large 2D motion and variability through the frames, due to large camera motion, large actor shape deformation, fast actor action and so on. In this work, we aim to study the performance of cuboid-aware feature aggregation in action detection under large action. Further, we propose to enhance actor fe…
The University of Tokyo（东京大学）；RIKEN（理化学研究所）；Dalian University of Technology（大连理工大学）；AIST（产业技术综合研究所）
Automated video-based assessment of surgical skills is a promising task in assisting young surgical trainees, especially in poor-resource areas. Existing works often resort to a CNN-LSTM joint framework that models long-term relationships by LSTMs on spatially pooled short-term CNN features. However, this practice would inevitably neglect the difference among semantic concepts such as tools, tissues, and background in the spatial dimension, impeding the subsequent temporal relationship modeling. In this paper, we propose a novel skill assessment framework, Video Semantic Aggregation (ViSA), wh…
First Place Solution to the CVPR'2023 AQTC Challenge: A Function-Interaction Centric Approach with Spatiotemporal Visual-Language Alignment
Hunan University（湖南大学）；Quanzhou Institute of Industrial Design and Machine Intelligence Innovation, Hunan University（湖南大学泉州工业设计与机器智能创新研究院）；University of Central Florida（中佛罗里达大学）
Tencent（腾讯）；Nanjing University（南京大学）；Southern University of Science and Technology（南方科技大学）
Temporal modeling is key for action recognition in videos. It normally considers both short-range motions and long-range aggregations. In this paper, we propose a Temporal Excitation and Aggregation (TEA) block, including a motion excitation (ME) module and a multiple temporal aggregation (MTA) module, specifically designed to capture both short- and long-range temporal evolution. In particular, for short-range motion modeling, the ME module calculates the feature-level temporal differences from spatiotemporal features. It then utilizes the differences to excite the motion-sensitive channels o…
Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）；Chongqing Institute for Brain and Intelligence（重庆脑科学与智能技术研究院）；Guangyang Bay Laboratory（广阳湾实验室）
As an emerging vision platform, a drone can look from many abnormal viewpoints which brings many new challenges into the classic vision task of video saliency prediction. To investigate these challenges, this paper proposes a large-scale video dataset for aerial saliency prediction, which consists of ground-truth salient object regions of 1,000 aerial videos, annotated by 24 subjects. To the best of our knowledge, it is the first large-scale video dataset that focuses on visual saliency prediction on drones. Based on this dataset, we propose a Model-guided Multi-path Network (MM-Net) that serv…
DeLiVR: Differential Spatiotemporal Lie Bias for Efficient Video Deraining
University of Chinese Academy of Sciences（中国科学院大学）；Qilu University of Technology（齐鲁工业大学）；Hubei University（湖北大学）；Nanjing University of Science and Technology（南京理工大学）；Shandong Normal University（山东师范大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation
School of Computer Engineering, Jimei University, Xiamen, China（厦门大学计算机工程学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China（南京航空航天大学人工智能学院）
CoST: Efficient Collaborative Perception From Unified Spatiotemporal Perspective
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；University of California, Los Angeles（美国加州大学洛杉矶分校）
Spatial-Temporal Decoupled Adapter for Micro-gesture Online Recognition
Hefei University of Technology（合肥工业大学）；United Arab Emirates University（阿联酋大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥国家综合科学中心）；Anhui Evolution Technology Co., Ltd.（安徽进化科技有限公司）
Temporal Reasoning is one important functionality for vision intelligence. In computer vision research community, temporal reasoning is usually studied in the form of video classification, for which many state-of-the-art Neural Network structures and dataset benchmarks are proposed in recent years, especially 3D CNNs and Kinetics. However, some recent works found that current video classification benchmarks contain strong biases towards static features, thus cannot accurately reflect the temporal modeling ability. New video classification benchmarks aiming to eliminate static biases are propos…
Huazhong University of Science and Technology（华中科技大学）；Columbia University（哥伦比亚大学）；Dalian University of Technology（大连理工大学）
This paper addresses the task of segmenting class-agnostic objects in semi-supervised setting. Although previous detection based methods achieve relatively good performance, these approaches extract the best proposal by a greedy strategy, which may lose the local patch details outside the chosen candidate. In this paper, we propose a novel spatiotemporal graph neural network (STG-Net) to reconstruct more accurate masks for video object segmentation, which captures the local contexts by utilizing all proposals. In the spatial graph, we treat object proposals of a frame as nodes and represent th…
Graph convolutional networks (GCNs) are widely adopted in skeleton-based action recognition due to their powerful ability to model data topology. We argue that the performance of recent proposed skeleton-based action recognition methods is limited by the following factors. First, the predefined graph structures are shared throughout the network, lacking the flexibility and capacity to model the multi-grain semantic information. Second, the relations among the global joints are not fully exploited by the graph local convolution, which may lose the implicit joint relevance. For instance, actions…
NeuroDETR: Multiscale Discrete Spatiotemporal Tokenization for Asynchronous EEG Event Detection
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）；Wuhan University（武汉大学）
Recent years have witnessed rapid advances in learnt video coding. Most algorithms have solely relied on the vector-based motion representation and resampling (e.g., optical flow based bilinear sampling) for exploiting the inter frame redundancy. In spite of the great success of adaptive kernel-based resampling (e.g., adaptive convolutions and deformable convolutions) in video prediction for uncompressed videos, integrating such approaches with rate-distortion optimization for inter frame coding has been less successful. Recognizing that each resampling solution offers unique advantages in reg…
Nanjing University（南京大学）；New York University（纽约大学）
Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses vari…
FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining
Shandong University（山东大学）；Central South University（中南大学）；Nanjing University of Science and Technology（南京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；National University of Defense Technology（国防科技大学）
Department of Systems Design Engineering（系统设计工程系）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；University of Waterloo（滑铁卢大学）
A core challenge faced by the majority of individuals with Autism Spectrum Disorder (ASD) is an impaired ability to infer other people's emotions based on their facial expressions. With significant recent advances in machine learning, one potential approach to leveraging technology to assist such individuals to better recognize facial expressions and reduce the risk of possible loneliness and depression due to social isolation is the design of computer vision-driven facial expression recognition systems. Motivated by this social need as well as the low latency requirement of such systems, this…
Rethinking the Readout: Unlocking Video Backbones for AI-Generated Video Detection
Huazhong University of Science and Technology（华中科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Jilin University（吉林大学）；Vast Intelligence Lab（旷视智能实验室）
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机科学学院）；Center for Data Science, Peking University（北京大学数据科学中心）
Spatially-guided Temporal Aggregation for Robust Event-RGB Optical Flow Estimation
Department of Automation, University of Science and Technology of China（自动化系，中国科学技术大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；Department of Mechanical Engineering, City University of Hong Kong（机械工程系，香港城市大学）；College of Computer Science, Beijing University of Technology（计算机科学学院，北京理工大学）
Lero-Irish Software Research Institute（Lero爱尔兰软件研究所）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；Insight Centre for Data Analytics（数据分析洞察中心）
Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph driven representation of video data. VEKG models video ob…
TASED-Net is a 3D fully-convolutional network architecture for video saliency detection. It consists of two building blocks: first, the encoder network extracts low-resolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information. As a result, a single prediction map is produced from an input clip of multiple frames. Frame-wise saliency maps can be predicted by applying TASED-Net in a sliding-window fashion to a video. The proposed approach assumes…
Ritsumeikan University（立命馆大学）；Zhejiang University（浙江大学）；Dalian university of technology（大连理工大学）；Zhejiang Lab（之江实验室）
Deep learning-based super-resolution (SR) techniques have generally achieved excellent performance in the computer vision field. Recently, it has been proven that three-dimensional (3D) SR for medical volumetric data delivers better visual results than conventional two-dimensional (2D) processing. However, deepening and widening 3D networks increases training difficulty significantly due to the large number of parameters and small number of training samples. Thus, we propose a 3D convolutional neural network (CNN) for SR of medical volumetric data called ParallelNet using parallel connections.…
Inception Institute of Artificial Intelligence (IIAI)（起源人工智能研究院）；Tencent YouTu Lab（腾讯优图实验室）
Video-based person re-identification (re-ID) is an important research topic in computer vision. The key to tackling the challenging task is to exploit both spatial and temporal clues in video sequences. In this work, we propose a novel graph-based framework, namely Multi-Granular Hypergraph (MGH), to pursue better representational capabilities by modeling spatiotemporal dependencies in terms of multiple granularities. Specifically, hypergraphs with different spatial granularities are constructed using various levels of part-based features across the video sequence. In each hypergraph, differen…
Automated interpretation of congenital heart disease from multi-view echocardiograms
Capital Medical University（首都医科大学）；Carnegie Mellon University（卡内基梅隆大学）；Harvard University（哈佛大学）；Harvard Medical School（哈佛医学院）；Beijing Children’s Hospital（北京儿童医院）；National Center for Children’s Health（国家儿童健康中心）；Harbin Engineering University（哈尔滨工程大学）；National Research Institute for Family Planning（国家计划生育委员会科学技术研究所）
