Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.
Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning
University of Strasbourg（斯特拉斯堡大学）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡图像引导外科研究所（IHU-Strasbourg））；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；Institute for Research Against Digestive Cancer (IRCAD)（消化道癌症研究所（IRCAD））
National University of Singapore（新加坡国立大学）；Singapore Institute for Neurotechnology（新加坡神经技术研究所）
In this paper, we address the challenging problem of action recognition, using event-based cameras. To recognise most gestural actions, often higher temporal precision is required for sampling visual information. Actions are defined by motion, and therefore, when using event-based cameras it is often unnecessary to re-sample the entire scene. Neuromorphic, event-based cameras have presented an alternative to visual information acquisition by asynchronously time-encoding pixel intensity changes, through temporally precise spikes (10 micro-second resolution), making them well equipped for action…
Stony Brook University（石溪大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
In this paper, we propose self-supervised training for video transformers using unlabeled video data. From a given video, we create local and global spatiotemporal views with varying spatial sizes and frame rates. Our self-supervised objective seeks to match the features of these different views representing the same video, to be invariant to spatiotemporal variations in actions. To the best of our knowledge, the proposed approach is the first to alleviate the dependency on negative samples or dedicated memory banks in Self-supervised Video Transformer (SVT). Further, owing to the flexibility …
Nanjing University（南京大学）；Shanghai AI Lab（上海人工智能实验室）
Traditional video action detectors typically adopt the two-stage pipeline, where a person detector is first employed to generate actor boxes and then 3D RoIAlign is used to extract actor-specific features for classification. This detection paradigm requires multi-stage training and inference, and cannot capture context information outside the bounding box. Recently, a few query-based action detectors are proposed to predict action instances in an end-to-end manner. However, they still lack adaptability in feature sampling and decoding, thus suffering from the issues of inferior performance or …
From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition
Aalborg University（奥尔堡大学）；Aarhus University（奥胡斯大学）；Poineer Centre for AI（先锋人工智能中心）；University of Southern Denmark（南丹麦大学）；Milestone System（Milestone系统公司）
DCL-SE: Dynamic Curriculum Learning for Spatiotemporal Encoding of Brain Imaging
University of Chinese Academy of Sciences（中国科学院大学）；Capital Medical University（首都医科大学）；Beijing Tongren Hospital（北京同仁医院）；Beijing Institute of Ophthalmology（北京眼科研究所）；Wannan Medical University（皖南医学院）
Beihang University（北京航空航天大学）；Beijing Institute of Technology（北京理工大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；City University of Hong Kong（香港城市大学）；University of Cambridge（剑桥大学）；Tsinghua University（清华大学）；Beijing Tongren Hospital, Capital Medical University（首都医科大学附属北京同仁医院）；Shandong University（山东大学）；Northeastern University（东北大学）
School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）
Correctly perceiving micro-expression is difficult since micro-expression is an involuntary, repressed, and subtle facial expression, and efficiently revealing the subtle movement changes and capturing the significant segments in a micro-expression sequence is the key to micro-expression recognition (MER). To handle the crucial issue, in this paper, we firstly propose a dynamic segmented sparse imaging module (DSSI) to compute dynamic images as local-global spatiotemporal descriptors under a unique sampling protocol, which reveals the subtle movement changes visually in an efficient way. Secon…
V2X-RECT: An Efficient V2X Trajectory Prediction Framework via Redundant Interaction Filtering and Tracking Error Correction
Communication Research Center, Harbin Institute of Technology（哈尔滨工业大学通信研究所）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）电子与信息工程学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
University of Oulu（奥卢大学）；École de Technologie Supérieure（高等技术学院）；VTT Technical Research Centre of Finland Ltd（芬兰VTT技术研究中心有限公司）
Video-based analysis of facial expressions has been increasingly applied to infer health states of individuals, such as depression and pain. Among the existing approaches, deep learning models composed of structures for multiscale spatiotemporal processing have shown strong potential for encoding facial dynamics. However, such models have high computational complexity, making for a difficult deployment of these solutions. To address this issue, we introduce a new technique to decompose the extraction of multiscale spatiotemporal features. Particularly, a building block structure called Decompo…
Department of Information Engineering Hong Kong；Department of Information Engineering
Since the wide employment of deep learning frameworks in video salient object detection, the accuracy of the recent approaches has made stunning progress. These approaches mainly adopt the sequential modules, based on optical flow or recurrent neural network (RNN), to learn robust spatiotemporal features. These modules are effective but significantly increase the computational burden of the corresponding deep models. In this paper, to simplify the network and maintain the accuracy, we present a lightweight network tailored for video salient object detection through the spatiotemporal knowledge…
EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息获取与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Secure Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial International Joint Research Center for Advanced Technology in Medical Imaging（安徽省医学影像先进技术国际联合研究中心）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Computer Vision Laboratory, University of Sassari（萨萨里大学计算机视觉实验室）
Unlike conventional frame-based sensors, event-based visual sensors output information through spikes at a high temporal resolution. By only encoding changes in pixel intensity, they showcase a low-power consuming, low-latency approach to visual information sensing. To use this information for higher sensory tasks like object recognition and tracking, an essential simplification step is the extraction and learning of features. An ideal feature descriptor must be robust to changes involving (i) local transformations and (ii) re-appearances of a local event pattern. To that end, we propose a nov…
DDTracking: A Deep Generative Framework for Diffusion MRI Tractography with Streamline Local-Global Spatiotemporal Modeling
University of Electronic Science and Technology of China（电子科技大学）；Nanjing University of Science and Technology（南京理工大学）；Brigham and Women’s Hospital（布里奇沃特医院）；Harvard Medical School（哈佛医学院）
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
The performance of video saliency estimation techniques has achieved significant advances along with the rapid development of Convolutional Neural Networks (CNNs). However, devices like cameras and drones may have limited computational capability and storage space so that the direct deployment of complex deep saliency models becomes infeasible. To address this problem, this paper proposes a dynamic saliency estimation approach for aerial videos via spatiotemporal knowledge distillation. In this approach, five components are involved, including two teachers, two students and the desired spatiot…
We present a novel method of integrating motion and appearance cues for foreground object segmentation in unconstrained videos. Unlike conventional methods encoding motion and appearance patterns individually, our method puts particular emphasis on their mutual assistance. Specifically, we propose using an interactively constrained encoding (ICE) scheme to incorporate motion and appearance patterns into a graph that leads to a spatiotemporal energy optimization. The reason of utilizing ICE is that both motion and appearance cues for the same target share underlying correlative structure, thus …
Motion blur is a known issue in photography, as it limits the exposure time while capturing moving objects. Extensive research has been carried to compensate for it. In this work, a computational imaging approach for motion deblurring is proposed and demonstrated. Using dynamic phase-coding in the lens aperture during the image acquisition, the trajectory of the motion is encoded in an intermediate optical image. This encoding embeds both the motion direction and extent by coloring the spatial blur of each object. The color cues serve as prior information for a blind deblurring process, implem…
We propose an action classification algorithm which uses Locality-constrained Linear Coding (LLC) to capture discriminative information of human body variations in each spatiotemporal subsequence of a video sequence. Our proposed method divides the input video into equally spaced overlapping spatiotemporal subsequences, each of which is decomposed into blocks and then cells. We use the Histogram of Oriented Gradient (HOG3D) feature to encode the information in each cell. We justify the use of LLC for encoding the block descriptor by demonstrating its superiority over Sparse Coding (SC). Our se…
What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Lassonde School of Engineering（拉桑德工程学院）；York University（约克大学）；Vector Institute（矢量研究院）；FAIR, Meta Superintelligence Labs（Meta超级智能实验室FAIR）；Mila（米拉研究院）；McGill University（麦吉尔大学）；Goodfire AI（Goodfire人工智能公司）
École de Technologie Supérieure, Université du Québec（魁北克大学高等技术学院）
Facial expressions are one of the most powerful ways for depicting specific patterns in human behavior and describing human emotional state. Despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot handle properly variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task even for humans. In this paper, we investigate the suitability of state-of-the-art deep learning architectures based on con…
An Interpretable Implicit-Based Approach for Modeling Local Spatial Effects: A Case Study of Global Gross Primary Productivity
College of Urban and Environmental Sciences, Peking University（北京大学城市与环境学院）；The Hong Kong Polytechnic University（香港理工大学）；School of Safety Science, Tsinghua University（清华大学安全科学学院）；College of Surveying and Geo-Informatics, Tongji University（同济大学测绘与地理信息学院）；School of Architecture and Urban Planning, Shenzhen University（深圳大学建筑与城市规划学院）
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）
NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Faculty of Electrical Engineering and Computing University of Zagreb（Zagreb大学电气工程与计算学院）；Faculty of Science, Radboud University（Radboud大学科学学院）
