Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition
Macau University of Science and Technology（澳门科技大学）；Alibaba Group（阿里巴巴集团）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）
Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, Fan Wang, Du Zhang, Zhen Lei, Hao Li, Rong Jin
Decoupling spatiotemporal representation refers to decomposing the spatial and temporal features into dimension-independent factors. Although previous RGB-D-based motion recognition methods have achieved promising performance through the tightly coupled multi-modal spatiotemporal representation, they still suffer from (i) optimization difficulty under small data setting due to the tightly spatiotemporal-entangled modeling;(ii) information redundancy as it usually contains lots of marginal information that is weakly relevant to classification; and (iii) low interaction between multi-modal spatiotemporal information caused by insufficient late fusion. To alleviate these drawbacks, we propose to decouple and recouple spatiotemporal representation for RGB-D-based motion recognition. Specifically, we disentangle the task of learning spatiotemporal representation into 3 sub-tasks: (1) Learning high-quality and dimension independent features through a decoupled spatial and temporal modeling network. (2) Recoupling the decoupled representation to establish stronger space-time dependency. (3) Introducing a Cross-modal Adaptive Posterior Fusion (CAPF) mechanism to capture cross-modal spatiotemporal information from RGB-D data. Seamless combination of these novel designs forms a robust spatialtemporal representation and achieves better performance than state-of-the-art methods on four public motion datasets. Our code is available at https://github.com/damo-cv/MotionRGBD.
This paper presents an effective method for generating a spatiotemporal (time-varying) texture map for a dynamic object using a single RGB-D camera. The input of our framework is a 3D template model and an RGB-D image sequence. Since there are invisible areas of the object at a frame in a single-camera setup, textures of such areas need to be borrowed from other frames. We formulate the problem as an MRF optimization and define cost functions to reconstruct a plausible spatiotemporal texture for a dynamic object. Experimental results demonstrate that our spatiotemporal textures can reproduce t…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
The key to address clothes-changing person re-identification (re-id) is to extract clothes-irrelevant features, e.g., face, hairstyle, body shape, and gait. Most current works mainly focus on modeling body shape from multi-modality information (e.g., silhouettes and sketches), but do not make full use of the clothes-irrelevant information in the original RGB images. In this paper, we propose a Clothes-based Adversarial Loss (CAL) to mine clothes-irrelevant features from the original RGB images by penalizing the predictive power of re-id model w.r.t. clothes. Extensive experiments demonstrate t…
EventFace: Event-Based Face Recognition via Structure-Driven Spatiotemporal Modeling
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息获取与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Secure Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial International Joint Research Center for Advanced Technology in Medical Imaging（安徽省医学影像先进技术国际联合研究中心）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Computer Vision Laboratory, University of Sassari（萨萨里大学计算机视觉实验室）
PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；College of Mechanical and Vehicle Engineering, Chongqing University（重庆大学机械与车辆工程学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Graph convolutional networks (GCNs) are widely adopted in skeleton-based action recognition due to their powerful ability to model data topology. We argue that the performance of recent proposed skeleton-based action recognition methods is limited by the following factors. First, the predefined graph structures are shared throughout the network, lacking the flexibility and capacity to model the multi-grain semantic information. Second, the relations among the global joints are not fully exploited by the graph local convolution, which may lose the implicit joint relevance. For instance, actions…
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
Depth can provide useful geographical cues for salient object detection (SOD), and has been proven helpful in recent RGB-D SOD methods. However, existing video salient object detection (VSOD) methods only utilize spatiotemporal information and seldom exploit depth information for detection. In this paper, we propose a depth-cooperated trimodal network, called DCTNet for VSOD, which is a pioneering work to incorporate depth information to assist VSOD. To this end, we first generate depth from RGB frames, and then propose an approach to treat the three modalities unequally. Specifically, a multi…
A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition
Macau University of Science and Technology（澳门科技大学）；DAMO Academy, Alibaba Group (U.S.) Inc.（阿里巴巴集团（美国）有限公司达摩院）；Amazon（亚马逊）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Capturing spatiotemporal dynamics is an essential topic in video recognition. In this paper, we present learnable higher-order operations as a generic family of building blocks for capturing spatiotemporal dynamics from RGB input video space. Similar to higher-order functions, the weights of higher-order operations are themselves derived from the data with learnable parameters. Classical architectures such as residual learning and network-in-network are first-order operations where weights are directly learned from the data. Higher-order operations make it easier to capture context-sensitive p…
The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）
Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach. The OFF is derived from the definition of optical flow and is orthogonal to the optical flow. The derivation also provides theoretical support for using the difference between two frames. By directly calculating pixel-wise spatiotemporal gradients of the deep feature maps, the OFF could b…
Istituto Italiano di Tecnologia（意大利技术研究院）；Universita‘ degli Studi di Genova（热那亚大学）；Universita‘ di Verona（维罗纳大学）
Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities could be available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This paper presents a…
MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence
University of Macau（澳门大学）；Xi'an Jiaotong University（西安交通大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；GVC Lab, Great Bay University（大湾大学GVC实验室）
SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；YUKUN Intelligent World（宇琨智能世界）
Strip-Fusion: Spatiotemporal Fusion for Multispectral Pedestrian Detection
Department of Mechanical Engineering, University of Michigan（机械工程系，密歇根大学）；Electrical Engineering and Computer Science Department, University of Michigan（电气工程与计算机科学系，密歇根大学）；Robotics Department, University of Michigan（机器人学系，密歇根大学）
Pain Analysis using Adaptive Hierarchical Spatiotemporal Dynamic Imaging
IEMN - DOAE UMR CNRS 8520；Université Polytechnique Hauts-de-France（上法兰西理工大学）；École de technologie supérieure（高等技术学院）；Université du Québec Montreal（魁北克大学蒙特利尔分校）；Sorbonne Center for Artificial Intelligence（索邦人工智能中心）
STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（北京大学深圳研究生院通用人工智能国家重点实验室）；University at Buffalo SUNY（纽约州立大学布法罗分校）；Department of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系）
4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling
Shanghai Jiaotong University（上海交通大学）；Zhongguancun Academy（中关村学院）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；Harbin Institute of Technology（哈尔滨工业大学）
JD Finance（京东金融）；Huawei Cloud（华为云）；Chinese Academy of Sciences（中国科学院）
Face anti-spoofing is significant to the security of face recognition systems. Previous works on depth supervised learning have proved the effectiveness for face anti-spoofing. Nevertheless, they only considered the depth as an auxiliary supervision in the single frame. Different from these methods, we develop a new method to estimate depth information from multiple RGB frames and propose a depth-supervised architecture which can efficiently encodes spatiotemporal information for presentation attack detection. It includes two novel modules: optical flow guided feature block (OFFB) and convolut…
Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Based on the information sources, these representations can be broadly categorized into two groups based on RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people b…
Spatially-guided Temporal Aggregation for Robust Event-RGB Optical Flow Estimation
Department of Automation, University of Science and Technology of China（自动化系，中国科学技术大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；Department of Mechanical Engineering, City University of Hong Kong（机械工程系，香港城市大学）；College of Computer Science, Beijing University of Technology（计算机科学学院，北京理工大学）
This paper presents a pure transformer-based approach, dubbed the Multi-Modal Video Transformer (MM-ViT), for video action recognition. Different from other schemes which solely utilize the decoded RGB frames, MM-ViT operates exclusively in the compressed video domain and exploits all readily available modalities, i.e., I-frames, motion vectors, residuals and audio waveform. In order to handle the large number of spatiotemporal tokens extracted from multiple modalities, we develop several scalable model variants which factorize self-attention across the space, time and modality dimensions. In …
The Chinese University of HongKong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；SenseTime Research（商汤科技研究院）；Nanyang Technological University（南洋理工大学）
Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convolutional networks (GCN) to extract features on top of human skeletons. Despite the positive results shown in previous works, GCN-based methods are subject to limitations in robustness, interoperability, and scalability. In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. Compare…
University at Buffalo（布法罗大学）；ETH Zürich（苏黎世联邦理工学院）；OPPO US Research Center（OPPO美国研究中心）；InnoPeak Tech（InnoPeak科技）；University of Tübingen（图宾根大学）
Visually exploring in a real-world 4D spatiotemporal space freely in VR has been a long-term quest. The task is especially appealing when only a few or even single RGB cameras are used for capturing the dynamic scene. To this end, we present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. First, we propose to decompose the 4D spatiotemporal space according to temporal characteristics. Points in the 4D space are associated with probabilities of belonging to three categories: static, deforming, and new areas. Each area is represented and regular…
Google（谷歌）；University of Michigan（密歇根大学）；Princeton University（普林斯顿大学）
State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by inc…
Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition
Beijing Engineering Research Center of Mixed Reality and Advanced Display（北京混合现实与先进显示工程研究中心）；School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院）
Representation-Centric Survey of Supervised Skeletal Action Recognition and the New Benchmark
School of Computing, Australian National University（澳大利亚国立大学计算学院）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；OPPO US Research Center（OPPO美国研究中心）；Carnegie Mellon University（卡内基梅隆大学）；University of Western Australia（西澳大利亚大学）；Curtin University（Curtin大学）；School of Engineering and Built Environment, Griffith University（格里菲斯大学工程与环境学院）；School of Medicine, Yale University（耶鲁大学医学院）
There is significant progress in recognizing traditional human activities from videos focusing on highly distinctive actions involving discriminative body movements, body-object and/or human-human interactions. Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. To address this, we propose a novel framework by exploiting the spatiotemporal attention to model the subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable g…
