本工作提出了一种端到端的多相机3D多目标跟踪（MOT）框架。该框架强调时空连续性，并结合了对被跟踪对象的过去和未来推理。因此，我们将其命名为“Past-and-Future reasoning for Tracking”（PF-Track）。具体而言，我们的方法采用了“tracking by attention”框架，并通过object queries随时间连贯地表示被跟踪实例。为了显式利用历史线索，我们的“Past Reasoning”模块通过交叉关注来自先前帧和其他对象的queries，学习优化轨迹并增强对象特征。“Future Reasoning”模块消化历史信息并预测鲁棒的未来轨迹。在长期遮挡的情况下，我们的方法通过整合运动预测来维持对象位置并实现重新关联。在nuScenes数据集上，我们的方法大幅提升了AMOTA，并与先前方法相比显著减少了90%的ID-Switches，这比之前低了一个数量级。代码和模型已在https://github.com/TRI-ML/PF-Track上提供。
英文摘要：
This work proposes an end-to-end multi-camera 3D multi-object tracking (MOT) framework. It emphasizes spatio-temporal continuity and integrates both past and future reasoning for tracked objects. Thus, we name it "Past-and-Future reasoning for Tracking" (PF-Track). Specifically, our method adapts the "tracking by attention" framework and represents tracked instances coherently over time with object queries. To explicitly use historical cues, our "Past Reasoning" module learns to refine the tracks and enhance the object features by cross-attending to queries from previous frames and other objects. The "Future Reasoning" module digests historical information and predicts robust future trajectories. In the case of long-term occlusions, our method maintains the object positions and enables re-association by integrating motion predictions. On the nuScenes dataset, our method improves AMOTA by a large margin and remarkably reduces ID-Switches by 90% compared to prior approaches, which is an order of magnitude less. The code and models are made available at https://github.com/TRI-ML/PF-Track.
Event camera has offered promising alternative for visual perception, especially in high speed and high dynamic range scenes. Recently, many deep learning methods have shown great success in providing promising solutions to many event-based problems, such as optical flow estimation. However, existing deep learning methods did not address the importance of temporal information well from the perspective of architecture design and cannot effectively extract spatio-temporal features. Another line of research that utilizes Spiking Neural Network suffers from training issues for deeper architecture.…
HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking
Sanya Science and Education Innovation Park, Wuhan University of Technology（武汉理工大学三亚科学教育创新园）；Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology（湖北省交通运输物联网重点实验室，计算机科学与人工智能学院，武汉理工大学）；State Key Laboratory for Multimedia Information Processing, Peking University（多媒体信息处理国家重点实验室，北京大学）
Kapital Bank（资本银行）；Khazar University（哈扎尔大学）；Consumer Experience Tribe（消费者体验部落）；Lumos Extension（Lumos扩展）；School of Data（数据学院）
Recognizing human actions based on videos has became one of the most popular areas of research in computer vision in recent years. This area has many applications such as surveillance, robotics, health care, video search and human-computer interaction. There are many problems associated with recognizing human actions in videos such as cluttered backgrounds, obstructions, viewpoints variation, execution speed and camera movement. A large number of methods have been proposed to solve the problems. This paper focus on spatial and temporal pattern recognition for the classification of videos using…
Nanjing University of Science and Technology（南京理工大学）；Yale-NUS College（耶鲁-新加坡国立大学学院）；National University of Singapore（新加坡国立大学）
Occlusion is a long-standing problem that causes many modern tracking methods to be erroneous. In this paper, we address the occlusion problem by exploiting the current and future possible locations of the target object from its past trajectory. To achieve this, we introduce a learning-based tracking method that takes into account background motion modeling and trajectory prediction. Our trajectory prediction module predicts the target object's locations in the current and future frames based on the object's past trajectory. Since, in the input video, the target object's trajectory is not only…
CowCare: A Unified Visual Framework for Viewpoint-Robust Cattle Tracking and Spatiotemporal Heat-Stress Analysis
College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）；National Innovation Center for Digital Fishery, Ministry of Agriculture and Rural Affairs（农业农村部国家数字渔业创新中心）；Key Laboratory of Smart Farming Technologies for Aquatic Animal and Livestock, Ministry of Agriculture and Rural Affairs（农业农村部水产动物与畜禽智慧养殖技术重点实验室）；Key Laboratory of Agricultural Informatization Standardization, Ministry of Agriculture and Rural Affairs（农业农村部农业信息化标准化重点实验室）
Spatio-Temporal State Space Model For Efficient Event-Based Optical Flow
Advanced Research and Innovation Center (ARIC), Khalifa University, Abu Dhabi, UAE（高级研究与创新中心（ARIC），哈利法大学，阿布扎比，阿联酋）；Department of Computer Science, Khalifa University of Science and Technology, Abu Dhabi, UAE（计算机科学系，哈利法科学技术大学，阿布扎比，阿联酋）
Camera and radar sensors have significant advantages in cost, reliability, and maintenance compared to LiDAR. Existing fusion methods often fuse the outputs of single modalities at the result-level, called the late fusion strategy. This can benefit from using off-the-shelf single sensor detection algorithms, but late fusion cannot fully exploit the complementary properties of sensors, thus having limited performance despite the huge potential of camera-radar fusion. Here we propose a novel proposal-level early fusion approach that effectively exploits both spatial and contextual properties of …
This paper presents an effective method for generating a spatiotemporal (time-varying) texture map for a dynamic object using a single RGB-D camera. The input of our framework is a 3D template model and an RGB-D image sequence. Since there are invisible areas of the object at a frame in a single-camera setup, textures of such areas need to be borrowed from other frames. We formulate the problem as an MRF optimization and define cost functions to reconstruct a plausible spatiotemporal texture for a dynamic object. Experimental results demonstrate that our spatiotemporal textures can reproduce t…
Hanyang University（汉阳大学）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
In this paper, we propose a new joint object detection and tracking (JoDT) framework for 3D object detection and tracking based on camera and LiDAR sensors. The proposed method, referred to as 3D DetecTrack, enables the detector and tracker to cooperate to generate a spatio-temporal representation of the camera and LiDAR data, with which 3D object detection and tracking are then performed. The detector constructs the spatio-temporal features via the weighted temporal aggregation of the spatial features obtained by the camera and LiDAR fusion. Then, the detector reconfigures the initial detecti…
University of Oregon（俄勒冈大学）；University of Virginia（弗吉尼亚大学）
Object re-identification (ReID) is a key application of city-scale cameras. While classic ReID tasks are often considered as image retrieval, we treat them as spatiotemporal queries for locations and times in which the target object appeared. Spatiotemporal reID is challenged by the accuracy limitation in computer vision algorithms and the colossal videos from city cameras. We present Clique, a practical ReID engine that builds upon two new techniques: (1) Clique assesses target occurrences by clustering fuzzy object features extracted by ReID algorithms, with each cluster representing the gen…
MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects
Human-Centered-Computing and Extended Reality Lab（以人为本计算与扩展现实实验室）；Klinikum rechts der Isar（右岸诊所）；Orthopedics and Sports Orthopedics（骨科和运动骨科）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所（MIRMI））；TH Köln（科隆大学）；Computer Graphics Group（计算机图形学组）；Photogrammetry and Remote Sensing Lab（摄影测量与遥感实验室）
Enhancing Multi-Camera People Tracking with Anchor-Guided Clustering and Spatio-Temporal Consistency ID Re-Assignment
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；National Center for High-Performance Computing（国家高性能计算中心）
提出一种结合锚定引导聚类与时空一致性的多摄像机多人员跟踪方法，在CVPR AI City Challenge 2023上以95.36% IDF1获第一名。
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）
3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design spatial cross-attention that each BEV query extracts th…
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for moving points captured in multiple unsynchronized videos and bundle adjustment is not designed to estimate the temporal alignment between cameras. We present a spatiotemporal bundle adjustment framework that jointly optimizes four coupled sub-problems: estimating camera intrinsics and extrinsics, triangulating static 3D points, as well as sub-frame temporal alignment between cameras and computing 3D trajectories of …
KIOS Research and Innovation Center of Excellence（KIOS研究与卓越创新中心）；University of Cyprus（塞浦路斯大学）
Autonomous vehicles increasingly rely on cameras to provide the input for perception and scene understanding and the ability of these models to classify their environment and objects, under adverse conditions and image noise is crucial. When the input is, either unintentionally or through targeted attacks, deteriorated, the reliability of autonomous vehicle is compromised. In order to mitigate such phenomena, we propose DriveGuard, a lightweight spatio-temporal autoencoder, as a solution to robustify the image segmentation process for autonomous vehicles. By first processing camera images with…
Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；The University of New South Wales（新南威尔士大学）
We present a novel method to reconstruct a spectral central view and its aligned disparity map from spatio-spectrally coded light fields. Since we do not reconstruct an intermediate full light field from the coded measurement, we refer to this as principal reconstruction. The coded light fields correspond to those captured by a light field camera in the unfocused design with a spectrally coded microlens array. In this application, the spectrally coded light field camera can be interpreted as a single-shot spectral depth camera. We investigate several multi-task deep learning methods and propos…
Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Computing, Department of Computer Science, National University of Singapore（新加坡国立大学计算机学院）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
MetaOcc: Spatio-Temporal Fusion of Surround-View 4D Radar and Camera for 3D Occupancy Prediction with Dual Training Strategies
School of Automotive Studies, Tongji University（同济大学汽车学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Automobile, Chang'an University（长安大学汽车学院）；School of Information and Electrical Engineering, Hangzhou City University（杭州城市学院信息与电气工程学院）；College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）
SIS-Challenge: Event-based Spatio-temporal Instance Segmentation Challenge at the CVPR 2025 Event-based Vision Workshop
TU Berlin, SCIoI, ECDF（柏林技术大学、SCIoI、ECDF）；RWTH Aachen（亚琛工业大学）；Xidian University（西安电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）；Wonkwang University（Wonkwang大学）；Tongji University（同济大学）
Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and recon…
Hunan University（湖南大学）；Technische Universität Berlin（柏林工业大学）；Einstein Center Digital Future（爱因斯坦数字未来中心）；Hong Kong University of Science and Technology（香港科技大学）
Identifying independently moving objects is an essential task for dynamic scene understanding. However, traditional cameras used in dynamic scenes may suffer from motion blur or exposure artifacts due to their sampling principle. By contrast, event-based cameras are novel bio-inspired sensors that offer advantages to overcome such limitations. They report pixelwise intensity changes asynchronously, which enables them to acquire visual information at exactly the same rate as the scene dynamics. We develop a method to identify independently moving objects acquired with an event-based camera, i.e…
Hebei University of Technology（河北工业大学）；Australian National University（澳大利亚国立大学）；Tencent（腾讯）
Single-image super-resolution (SR) and multi-frame SR are two ways to super resolve low-resolution images. Single-Image SR generally handles each image independently, but ignores the temporal information implied in continuing frames. Multi-frame SR is able to model the temporal dependency via capturing motion information. However, it relies on neighbouring frames which are not always available in the real world. Meanwhile, slight camera shake easily causes heavy motion blur on long-distance-shot low-resolution images. To address these problems, a Blind Motion Deblurring Super-Reslution Network…
We present new theoretical foundations for unsupervised Spike-Timing-Dependent Plasticity (STDP) learning in spiking neural networks (SNNs). In contrast to empirical parameter search used in most previous works, we provide novel theoretical grounds for SNN and STDP parameter tuning which considerably reduces design time. Using our generic framework, we propose a class of global, action-based and convolutional SNN-STDP architectures for learning spatio-temporal features from event-based cameras. We assess our methods on the N-MNIST, the CIFAR10-DVS and the IBM DVS128 Gesture datasets, all acqui…
