事件相机是一种神经形态视觉传感器，将场景记录为稀疏且异步的事件流。大多数基于事件的方法将事件投影为稠密帧，并使用传统视觉模型进行处理，导致计算复杂度较高。近期趋势是开发基于点的网络，通过学习稀疏表示实现高效的事件处理。然而，现有工作可能缺乏鲁棒的局部信息聚合器和有效的特征交互操作，从而限制了其建模能力。为此，我们提出了一种注意力感知模型，名为事件体素集合Transformer（Event Voxel Set Transformer，EVSTr），用于在事件流上进行高效的时空表示学习。它首先将事件流转换为体素集合，然后分层聚合体素特征以获得鲁棒的表示。EVSTr的核心是一个事件体素Transformer编码器，由两个精心设计的组件组成，包括用于局部信息聚合的多尺度邻居嵌入层（Multi-Scale Neighbor Embedding Layer，MNEL）和用于全局特征交互的体素自注意力层（Voxel Self-Attention Layer，VSAL）。为了使网络能够纳入长程时间结构，我们引入了一种分段建模策略（S$^{2}$TM），从一系列分段的体素集合中学习运动模式。所提出的模型在两个识别任务上进行了评估，包括物体分类和动作识别。为了提供有说服力的模型评估，我们提出了一个新的基于事件的动作识别数据集（NeuroHAR），该数据集在具有挑战性的场景中记录。综合实验表明，EVSTr在保持较低模型复杂度的同时，取得了最先进的性能。
英文摘要：
Event cameras are neuromorphic vision sensors that record a scene as sparse and asynchronous event streams. Most event-based methods project events into dense frames and process them using conventional vision models, resulting in high computational complexity. A recent trend is to develop point-based networks that achieve efficient event processing by learning sparse representations. However, existing works may lack robust local information aggregators and effective feature interaction operations, thus limiting their modeling capabilities. To this end, we propose an attention-aware model named Event Voxel Set Transformer (EVSTr) for efficient spatiotemporal representation learning on event streams. It first converts the event stream into voxel sets and then hierarchically aggregates voxel features to obtain robust representations. The core of EVSTr is an event voxel transformer encoder that consists of two well-designed components, including the Multi-Scale Neighbor Embedding Layer (MNEL) for local information aggregation and the Voxel Self-Attention Layer (VSAL) for global feature interaction. Enabling the network to incorporate a long-range temporal structure, we introduce a segment modeling strategy (S$^{2}$TM) to learn motion patterns from a sequence of segmented voxel sets. The proposed model is evaluated on two recognition tasks, including object classification and action recognition. To provide a convincing model evaluation, we present a new event-based action recognition dataset (NeuroHAR) recorded in challenging scenarios. Comprehensive experiments show that EVSTr achieves state-of-the-art performance while maintaining low model complexity.
This paper considers the problem of localizing actions in videos as a sequences of bounding boxes. The objective is to generate action proposals that are likely to include the action of interest, ideally achieving high recall with few proposals. Our contributions are threefold. First, inspired by selective search for object proposals, we introduce an approach to generate action proposals from spatiotemporal super-voxels in an unsupervised manner, we call them Tubelets. Second, along with the static features from individual frames our approach advantageously exploits motion. We introduce indepe…
Understanding the content of videos is one of the core techniques for developing various helpful applications in the real world, such as recognizing various human actions for surveillance systems or customer behavior analysis in an autonomous shop. However, understanding the content or story of the video still remains a challenging problem due to its sheer amount of data and temporal structure. In this paper, we propose a multi-channel neural network structure that adopts a two-stream network structure, which has been shown high performance in human action recognition field, and use it as a sp…
Graz University of Technology（格拉茨技术大学）；York University（约克大学）
Two-stream Convolutional Networks (ConvNets) have shown strong performance for human action recognition in videos. Recently, Residual Networks (ResNets) have arisen as a new technique to train extremely deep architectures. In this paper, we introduce spatiotemporal ResNets as a combination of these two approaches. Our novel architecture generalizes ResNets for the spatiotemporal domain by introducing residual connections in two ways. First, we inject residual connections between the appearance and motion pathways of a two-stream architecture to allow spatiotemporal interaction between the two …
Indiana University, Bloomington（印第安纳大学伯明顿分校）；University of Central Florida（中佛罗里达大学）；Midea Corporate Research Center（美的企业研究中心）
A major emerging challenge is how to protect people's privacy as cameras and computer vision are increasingly integrated into our daily lives, including in smart devices inside homes. A potential solution is to capture and record just the minimum amount of information needed to perform a task of interest. In this paper, we propose a fully-coupled two-stream spatiotemporal architecture for reliable human action recognition on extremely low resolution (e.g., 12x16 pixel) videos. We provide an efficient method to extract spatial and temporal features and to aggregate them into a robust feature re…
Tsinghua University（清华大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid netw…
This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary tim…
Zhejiang University（浙江大学）；SenseTime Group Limited（商汤集团有限公司）
Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in…
Environment perception including detection, classification, tracking, and motion prediction are key enablers for automated driving systems and intelligent transportation applications. Fueled by the advances in sensing technologies and machine learning techniques, LiDAR-based sensing systems have become a promising solution. The current challenges of this solution are how to effectively combine different perception tasks into a single backbone and how to efficiently learn the spatiotemporal features directly from point cloud sequences. In this research, we propose a novel spatiotemporal attenti…
IHPC, A*STAR（新加坡科技研究局高性能计算研究所）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Megvii Research（旷视研究院）；Nanjing Megvii Technology（南京旷视科技）
To facilitate depth-based 3D action recognition, 3D dynamic voxel (3DV) is proposed as a novel 3D motion representation. With 3D space voxelization, the key idea of 3DV is to encode 3D motion information within depth video into a regular voxel set (i.e., 3DV) compactly, via temporal rank pooling. Each available 3DV voxel intrinsically involves 3D spatial and motion feature jointly. 3DV is then abstracted as a point set and input into PointNet++ for 3D action recognition, in the end-to-end learning way. The intuition for transferring 3DV into the point set form is that, PointNet++ is lightweigh…
Graz University of Technology（格拉茨技术大学）；University of Oxford（牛津大学）
Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse su…
Spatiotemporal action localization requires the incorporation of two sources of information into the designed architecture: (1) temporal information from the previous frames and (2) spatial information from the key frame. Current state-of-the-art approaches usually extract these information with separate networks and use an extra mechanism for fusion to get detections. In this work, we present YOWO, a unified CNN architecture for real-time spatiotemporal action localization in video streams. YOWO is a single-stage architecture with two branches to extract temporal and spatial information concu…
York University（约克大学）；Vector Institute for AI（向量人工智能研究所）；Ryerson University（瑞尔森大学）；University of Guelph（圭尔夫大学）；Samsung AI Centre Toronto（三星多伦多人工智能中心）
Deep spatiotemporal models are used in a variety of computer vision tasks, such as action recognition and video object segmentation. Currently, there is a limited understanding of what information is captured by these models in their intermediate representations. For example, while it has been observed that action recognition algorithms are heavily influenced by visual appearance in single static frames, there is no quantitative methodology for evaluating such static bias in the latent representation compared to bias toward dynamic information (e.g. motion). We tackle this challenge by proposi…
Peking University（北京大学）；Chinese University of Hong Kong（香港中文大学）
We revisit Semantic Scene Completion (SSC), a useful task to predict the semantic and occupancy representation of 3D scenes, in this paper. A number of methods for this task are always based on voxelized scene representations for keeping local scene structure. However, due to the existence of visible empty voxels, these methods always suffer from heavy computation redundancy when the network goes deeper, and thus limit the completion quality. To address this dilemma, we propose our novel point-voxel aggregation network for this task. Firstly, we transfer the voxelized scenes to point clouds by…
Adversarial Attack for RGB-Event based Visual Object Tracking
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Beijing Institute of Technology（北京理工大学）；Zhejiang Normal University（浙江师范大学）；Peng Cheng Laboratory（鹏城实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室）；School of Electronic and Computer Engineering, Shenzhen, Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
Google（谷歌）；University of Michigan（密歇根大学）；Princeton University（普林斯顿大学）
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by inc…
Most current pipelines for spatio-temporal action localization connect frame-wise or clip-wise detection results to generate action proposals, where only local information is exploited and the efficiency is hindered by dense per-frame localization. In this paper, we propose Coarse-to-Fine Action Detector (CFAD),an original end-to-end trainable framework for efficient spatio-temporal action localization. The CFAD introduces a new paradigm that first estimates coarse spatio-temporal action tubes from video streams, and then refines the tubes' location based on key timestamps. This concept is imp…
Infrared (IR) imaging has the potential to enable more robust action recognition systems compared to visible spectrum cameras due to lower sensitivity to lighting conditions and appearance variability. While the action recognition task on videos collected from visible spectrum imaging has received much attention, action recognition in IR videos is significantly less explored. Our objective is to exploit imaging data in this modality for the action recognition task. In this work, we propose a novel two-stream 3D convolutional neural network (CNN) architecture by introducing the discriminative c…
This paper presents a spatiotemporal deep learning approach for mouse behavioural classification in the home-cage. Using a series of dual-stream architectures with assorted modifications to increase performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. To investigate the efficacy of this approach, models were evaluated by dissociating the streams and training/testing in the same rigorous manner as the main classifiers. Using an annotated, publicly available dataset of a singly-housed mice, we achieve predictio…
Tsinghua University（清华大学）；Beijing Innovation Center for Future Chip（北京未来芯片创新中心）
Super-resolution (SR) is a useful technology to generate a high-resolution (HR) visual output from the low-resolution (LR) visual inputs overcoming the physical limitations of the cameras. However, SR has not been applied to enhance the resolution of spatiotemporal event-stream images captured by the frame-free dynamic vision sensors (DVSs). SR of event-stream image is fundamentally different from existing frame-based schemes since basically each pixel value of DVS images is an event sequence. In this work, a two-stage scheme is proposed to solve the SR problem of the spatiotemporal event-stre…
EEvAct: Early Event-Based Action Recognition with High-Rate Two-Stream Spiking Neural Networks
fortiss GmbH（fortiss公司）；Technical University Munich（慕尼黑技术大学）；Neuromorphic Computing（类脑计算）；Simi Reality Motion Systems GmbH（Simi现实运动系统公司）；Digital Integrated Systems Group（数字集成系统组）
University of Alcalá（阿尔卡拉大学）；York University（约克大学）
In highway scenarios, an alert human driver will typically anticipate early cut-in and cut-out maneuvers of surrounding vehicles using only visual cues. An automated system must anticipate these situations at an early stage too, to increase the safety and the efficiency of its performance. To deal with lane-change recognition and prediction of surrounding vehicles, we pose the problem as an action recognition/prediction problem by stacking visual cues from video cameras. Two video action recognition approaches are analyzed: two-stream convolutional networks and spatiotemporal multiplier networ…
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institution（佐治亚理工研究所）
Recent two-stream deep Convolutional Neural Networks (ConvNets) have made significant progress in recognizing human actions in videos. Despite their success, methods extending the basic two-stream ConvNet have not systematically explored possible network architectures to further exploit spatiotemporal dynamics within video sequences. Further, such networks often use different baseline two-stream networks. Therefore, the differences and the distinguishing factors between various methods using Recurrent Neural Networks (RNN) or convolutional networks on temporally-constructed feature vectors (Te…
Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Querying event patterns from video streams is challenging due to its unstructured representation. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to query video streams due to their unstructured data model and lack of expressive query language. In this work, we focus on a CEP framework where users can define high-level expressive queries over videos to de…
Hiroshima University（广岛大学）；Prefectural University of Hiroshima（广岛县立大学）；University of Genoa（热那亚大学）；Istituto G.Gaslini Pediatric Hospital（G.加斯利尼儿科研究所）；Italian Institute of Technology（意大利技术研究院）
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders. However, its evaluation in clinical practice relies on visual inspection by experts, and an automated solution is eagerly awaited. Recently, video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information, such as background clutter in the video. Furthermore, for reliability, it is necessary to properly extract the spatiotemporal features of infants during GMs. In this study, we propose an automated GMs clas…
Tianjin University（天津大学）；University at Albany, SUNY（纽约州立大学奥尔巴尼分校）
The structured time series (STS) classification problem requires the modeling of interweaved spatiotemporal dependency. most previous STS classification methods model the spatial and temporal dependencies independently. Due to the complexity of the STS data, we argue that a desirable STS classification method should be a holistic framework that can be made as adaptive and flexible as possible. This motivates us to design a deep neural network with such merits. Inspired by the dual-stream hypothesis in neural science, we propose a novel dual-stream framework for modeling the interweaved spatiot…
The goal of convective storm nowcasting is local prediction of severe and imminent convective storms. Here, we consider the convective storm nowcasting problem from the perspective of machine learning. First, we use a pixel-wise sampling method to construct spatiotemporal features for nowcasting, and flexibly adjust the proportions of positive and negative samples in the training set to mitigate class-imbalance issues. Second, we employ a concise two-stream convolutional neural network to extract spatial and temporal cues for nowcasting. This simplifies the network structure, reduces the train…
PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；College of Mechanical and Vehicle Engineering, Chongqing University（重庆大学机械与车辆工程学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Evaluation of Key Spatiotemporal Learners for Print Track Anomaly Classification Using Melt Pool Image Streams
McGill University（麦吉尔大学）；National Research Council Canada（加拿大国家研究委员会）；Braintoy AI；Apollo Machine & Welding Ltd（阿波罗机器焊接有限公司）；Fraunhofer Institute for Laser Technology ILT（弗劳恩霍夫激光技术研究所ILT）；BCT GmbH（BCT有限公司）
V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation
State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；National Eng. Research Ctr. of Visual Technology, School of Computer Science, Peking University（北京大学计算机学院国家视觉技术工程研究中心）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；National Institute of Informatics（国立情报学研究所）
Carnegie Mellon University（卡内基梅隆大学）；Instituto Superior Técnico（高等技术学院）
Action detection and recognition tasks have been the target of much focus in the computer vision community due to their many applications, namely, security, robotics and recommendation systems. Recently, datasets like AVA, provide multi-person, multi-label, spatiotemporal action detection and recognition challenges. Being unable to discern which portions of the input to use for classification is a limitation of two-stream CNN approaches, once the vision task involves several people with several labels. We address this limitation and improve the state-of-the-art performance of two-stream CNNs. …
Beijing University of Technology（北京工业大学）；Central China Normal University（华中师范大学）；City University of Hong Kong（香港城市大学）；Southeast University（东南大学）
Event cameras attract researchers' attention due to their low power consumption, high dynamic range, and extremely high temporal resolution. Learning models on event-based object classification have recently achieved massive success by accumulating sparse events into dense frames to apply traditional 2D learning methods. Yet, these approaches necessitate heavy-weight models and are with high computational complexity due to the redundant information introduced by the sparse-to-dense conversion, limiting the potential of event cameras on real-life applications. This study aims to address the cor…
Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?
ECE, Seoul National University（首尔大学电气与计算机工程系）；IPAI, Seoul National University（首尔大学IPAI研究所）；BCS, Seoul National University（首尔大学生物与认知科学系）；Psychology, Seoul National University（首尔大学心理学系）；ASRI / INMC / AIIS, Seoul National University（首尔大学ASRI/INMC/AIIS研究所）
In this paper, we propose Two-Stream AMTnet, which leverages recent advances in video-based action representation[1] and incremental action tube generation[2]. Majority of the present action detectors follow a frame-based representation, a late-fusion followed by an offline action tube building steps. These are sub-optimal as: frame-based features barely encode the temporal relations; late-fusion restricts the network to learn robust spatiotemporal features; and finally, an offline action tube generation is not suitable for many real-world problems such as autonomous driving, human-robot inter…
Lero-Irish Software Research Institute（Lero爱尔兰软件研究所）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；Insight Centre for Data Analytics（数据分析洞察中心）
Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph driven representation of video data. VEKG models video ob…
Pacific Northwest National Laboratory（太平洋西北国家实验室）；Drexel University（德雷塞尔大学）；National Research Council of Canada（加拿大国家研究委员会）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
The Locally Competitive Algorithm (LCA) uses local competition between non-spiking leaky integrator neurons to infer sparse representations, allowing for potentially real-time execution on massively parallel neuromorphic architectures such as Intel's Loihi processor. Here, we focus on the problem of inferring sparse representations from streaming video using dictionaries of spatiotemporal features optimized in an unsupervised manner for sparse reconstruction. Non-spiking LCA has previously been used to achieve unsupervised learning of spatiotemporal dictionaries composed of convolutional kerne…
