Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Querying event patterns from video streams is challenging due to its unstructured representation. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to query video streams due to their unstructured data model and lack of expressive query language. In this work, we focus on a CEP framework where users can define high-level expressive queries over videos to detect a range of spatiotemporal event patterns. In this context, we propose: i) VidCEP, an in-memory, on the fly, near real-time complex event matching framework for video streams. The system uses a graph-based event representation for video streams which enables the detection of high-level semantic concepts from video using cascades of Deep Neural Network models, ii) a Video Event Query language (VEQL) to express high-level user queries for video streams in CEP, iii) a complex event matcher to detect spatiotemporal video event patterns by matching expressive user queries over video data. The proposed approach detects spatiotemporal video event patterns with an F-score ranging from 0.66 to 0.89. VidCEP maintains near real-time performance with an average throughput of 70 frames per second for 5 parallel videos with sub-second matching latency.
Hiroshima University（广岛大学）；Prefectural University of Hiroshima（广岛县立大学）；University of Genoa（热那亚大学）；Istituto G.Gaslini Pediatric Hospital（G.加斯利尼儿科研究所）；Italian Institute of Technology（意大利技术研究院）
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders. However, its evaluation in clinical practice relies on visual inspection by experts, and an automated solution is eagerly awaited. Recently, video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information, such as background clutter in the video. Furthermore, for reliability, it is necessary to properly extract the spatiotemporal features of infants during GMs. In this study, we propose an automated GMs clas…
This paper presents a spatiotemporal deep learning approach for mouse behavioural classification in the home-cage. Using a series of dual-stream architectures with assorted modifications to increase performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. To investigate the efficacy of this approach, models were evaluated by dissociating the streams and training/testing in the same rigorous manner as the main classifiers. Using an annotated, publicly available dataset of a singly-housed mice, we achieve predictio…
Tsinghua University（清华大学）；Beijing Innovation Center for Future Chip（北京未来芯片创新中心）
Super-resolution (SR) is a useful technology to generate a high-resolution (HR) visual output from the low-resolution (LR) visual inputs overcoming the physical limitations of the cameras. However, SR has not been applied to enhance the resolution of spatiotemporal event-stream images captured by the frame-free dynamic vision sensors (DVSs). SR of event-stream image is fundamentally different from existing frame-based schemes since basically each pixel value of DVS images is an event sequence. In this work, a two-stage scheme is proposed to solve the SR problem of the spatiotemporal event-stre…
Understanding the content of videos is one of the core techniques for developing various helpful applications in the real world, such as recognizing various human actions for surveillance systems or customer behavior analysis in an autonomous shop. However, understanding the content or story of the video still remains a challenging problem due to its sheer amount of data and temporal structure. In this paper, we propose a multi-channel neural network structure that adopts a two-stream network structure, which has been shown high performance in human action recognition field, and use it as a sp…
Graz University of Technology（格拉茨技术大学）；York University（约克大学）
Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two …
Indiana University, Bloomington（印第安纳大学伯明顿分校）；University of Central Florida（中佛罗里达大学）；Midea Corporate Research Center（美的企业研究中心）
A major emerging challenge is how to protect people's privacy as cameras and computer vision are increasingly integrated into our daily lives, including in smart devices inside homes. A potential solution is to capture and record just the minimum amount of information needed to perform a task of interest. In this paper, we propose a fully-coupled two-stream spatiotemporal architecture for reliable human action recognition on extremely low resolution (e.g., 12x16 pixel) videos. We provide an efficient method to extract spatial and temporal features and to aggregate them into a robust feature re…
The goal of convective storm nowcasting is local prediction of severe and imminent convective storms. Here, we consider the convective storm nowcasting problem from the perspective of machine learning. First, we use a pixel-wise sampling method to construct spatiotemporal features for nowcasting, and flexibly adjust the proportions of positive and negative samples in the training set to mitigate class-imbalance issues. Second, we employ a concise two-stream convolutional neural network to extract spatial and temporal cues for nowcasting. This simplifies the network structure, reduces the train…
Lero-Irish Software Research Institute（Lero爱尔兰软件研究所）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；Insight Centre for Data Analytics（数据分析洞察中心）
Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph driven representation of video data. VEKG models video ob…
Evaluation of Key Spatiotemporal Learners for Print Track Anomaly Classification Using Melt Pool Image Streams
McGill University（麦吉尔大学）；National Research Council Canada（加拿大国家研究委员会）；Braintoy AI；Apollo Machine & Welding Ltd（阿波罗机器焊接有限公司）；Fraunhofer Institute for Laser Technology ILT（弗劳恩霍夫激光技术研究所ILT）；BCT GmbH（BCT有限公司）
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institution（佐治亚理工研究所）
Recent two-stream deep Convolutional Neural Networks (ConvNets) have made significant progress in recognizing human actions in videos. Despite their success, methods extending the basic two-stream ConvNet have not systematically explored possible network architectures to further exploit spatiotemporal dynamics within video sequences. Further, such networks often use different baseline two-stream networks. Therefore, the differences and the distinguishing factors between various methods using Recurrent Neural Networks (RNN) or convolutional networks on temporally-constructed feature vectors (Te…
Carnegie Mellon University（卡内基梅隆大学）；Instituto Superior Técnico（高等技术学院）
Action detection and recognition tasks have been the target of much focus in the computer vision community due to their many applications, namely, security, robotics and recommendation systems. Recently, datasets like AVA, provide multi-person, multi-label, spatiotemporal action detection and recognition challenges. Being unable to discern which portions of the input to use for classification is a limitation of two-stream CNN approaches, once the vision task involves several people with several labels. We address this limitation and improve the state-of-the-art performance of two-stream CNNs. …
Tsinghua University（清华大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid netw…
This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary tim…
Zhejiang University（浙江大学）；SenseTime Group Limited（商汤集团有限公司）
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in…
Tianjin University（天津大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）
The structured time series (STS) classification problem requires the modeling of interweaved spatiotemporal dependency. most previous STS classification methods model the spatial and temporal dependencies independently. Due to the complexity of the STS data, we argue that a desirable STS classification method should be a holistic framework that can be made as adaptive and flexible as possible. This motivates us to design a deep neural network with such merits. Inspired by the dual-stream hypothesis in neural science, we propose a novel dual-stream framework for modeling the interweaved spatiot…
PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；College of Mechanical and Vehicle Engineering, Chongqing University（重庆大学机械与车辆工程学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Huazhong University of Science and Technology（华中科技大学）
Neuromorphic vision sensors (event cameras) are inherently suitable for spiking neural networks (SNNs) and provide novel neuromorphic vision data for this biomimetic model. Due to the spatiotemporal characteristics, novel data augmentations are required to process the unconventional visual signals of these cameras. In this paper, we propose a novel Event SpatioTemporal Fragments (ESTF) augmentation method. It preserves the continuity of neuromorphic data by drifting or inverting fragments of the spatiotemporal event stream to simulate the disturbance of brightness variations, leading to more r…
Infrared (IR) imaging has the potential to enable more robust action recognition systems compared to visible spectrum cameras due to lower sensitivity to lighting conditions and appearance variability. While the action recognition task on videos collected from visible spectrum imaging has received much attention, action recognition in IR videos is significantly less explored. Our objective is to exploit imaging data in this modality for the action recognition task. In this work, we propose a novel two-stream 3D convolutional neural network (CNN) architecture by introducing the discriminative c…
Graz University of Technology（格拉茨技术大学）；University of Oxford（牛津大学）
Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse su…
Understanding actions and gestures in video streams requires temporal reasoning of the spatial content from different time instants, i.e., spatiotemporal (ST) modeling. In this survey paper, we have made a comparative analysis of different ST modeling techniques for action and gecture recognition tasks. Since Convolutional Neural Networks (CNNs) are proved to be an effective tool as a feature extractor for static images, we apply ST modeling techniques on the features of static images from different time instants extracted by CNNs. All techniques are trained end-to-end together with a CNN feat…
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not designed to estimate the temporal alignment between cameras. We present a spatiotemporal bundle adjustment framework that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating static 3D points, as well as sub-frame temporal alignment between cameras and computing 3D trajectories of …
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Drexel University（德雷塞尔大学）；National Research Council of Canada（加拿大国家研究委员会）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
The Locally Competitive Algorithm (LCA) uses local competition between non-spiking leaky integrator neurons to infer sparse representations, allowing for potentially real-time execution on massively parallel neuromorphic architectures such as Intel's Loihi processor. Here, we focus on the problem of inferring sparse representations from streaming video using dictionaries of spatiotemporal features optimized in an unsupervised manner for sparse reconstruction. Non-spiking LCA has previously been used to achieve unsupervised learning of spatiotemporal dictionaries composed of convolutional kerne…
Google（谷歌）；University of Michigan（密歇根大学）；Princeton University（普林斯顿大学）
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by inc…
Asynchronous Event Stream Noise Filtering for High-frequency Structure Deformation Measurement
College of Aerospace Science and Engineering, National University of Defense Technology（航空航天科学与工程学院，国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（图像测量与视觉导航省级重点实验室）
Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed Rekall, a library th…
Microsoft Autonomous Systems and Robotics Research（微软自主系统与机器人研究部）；Microsoft Research AI4Science（微软研究院AI4Science）
Partial differential equations (PDEs) are central to describing complex physical system simulations. Their expensive solution techniques have led to an increased interest in deep neural network based surrogates. However, the practical utility of training such surrogates is contingent on their ability to model complex multi-scale spatio-temporal phenomena. Various neural network architectures have been proposed to target such phenomena, most notably Fourier Neural Operators (FNOs), which give a natural handle over local & global spatial information via parameterization of different Fourier mode…
Video anomaly detection aims to find the events in a video that do not conform to the expected behavior. The prevalent methods mainly detect anomalies by snippet reconstruction or future frame prediction error. However, the error is highly dependent on the local context of the current snippet and lacks the understanding of normality. To address this issue, we propose to detect anomalous events not only by the local context, but also according to the consistency between the testing event and the knowledge about normality from the training data. Concretely, we propose a novel two-stream framewor…
EventFlash: Towards Efficient MLLMs for Event-Based Vision
Xidian University（西安电子科技大学）；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)（广东人工智能与数字经济实验室（深圳））
StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；Nanjing University（南京大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation
School of Computer Engineering, Jimei University, Xiamen, China（厦门大学计算机工程学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China（南京航空航天大学人工智能学院）
Towards Video Anomaly Detection from Event Streams: A Baseline and Benchmark Datasets
Northwest Polytechnical University（西北工业大学）；School of Computing and Information Systems, Singapore Management University（新加坡国立大学计算机与信息系统学院）；Xidian University（西安电子科技大学）
EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息获取与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Secure Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial International Joint Research Center for Advanced Technology in Medical Imaging（安徽省医学影像先进技术国际联合研究中心）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Computer Vision Laboratory, University of Sassari（萨萨里大学计算机视觉实验室）
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院（教育部人工智能重点实验室））；Huawei Technologies（华为技术有限公司）
Highly Efficient 3D Human Pose Tracking from Events with Spiking Spatiotemporal Transformer
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（计算机科学与控制工程学院，深圳大学）；Department of Electrical and Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）
EventVL: Understand Event Streams via Multimodal Large Language Model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；KU Leuven（根特大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Carleton University（卡尔顿大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
In skeleton-based action recognition, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have achieved remarkable performance. However, in existing GCN-based methods, the topology of the graph is set manually, and it is fixed over all layers and input samples. This may not be optimal for the hierarchical GCN and diverse samples in action recognition tasks. In addition, the second-order information (the lengths and directions of bones) of the skeleton data, which is naturally more informative and discriminative for action recognition, is rarely i…
Spatiotemporal Attention Learning Framework for Event-Driven Object Recognition
City University of Hong Kong（香港城市大学）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）；State Key Laboratory of Terahertz and Millimeter Waves, City University of Hong Kong（香港城市大学太赫兹与毫米波国家重点实验室）
