Geometric-Topological Perception and Motion Prior for Real-Time Satellite Video Object Tracking
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Chinese Academy of Sciences（中国科学院地理空间信息处理与应用系统技术重点实验室）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子、电气与通信工程学院）
Satellite video object tracking (SVOT) remains fundamentally challenging due to texture scarcity, arbitrary rotation, aspect ratio changes, and severe occlusions. While recent state-of-the-art trackers excel in general scenarios, their reliance on rich appearance details or rigid spatial matching mechanisms leads to significant performance degradation in the satellite domain. To bridge this gap, we propose SiamGM, a real-time spatial-temporal unified tracking framework tailored for satellite videos. Instead of conventionally stacking modules, we synergize geometric-topological perception with temporal-kinematic prior, addressing the core limitations of SVOT. Spatially, we closely couple a Topological Attention Module (TAM) with a Geometry-Constrained Label Assignment (GCLA) method during the training phase, where a Local Graph Propagation (LGP) mechanism enforces neighborhood-consistent template-search correspondences. This joint paradigm establishes fine-grained structural correspondences and explicitly suppresses surrounding background noise. Temporally, rather than passively relying on visual cues, we propose an Online Motion Model Refinement (OMMR) strategy during the tracking phase. Gated by an online self-calibrated confidence indicator, OMMR adaptively fuse long-term stable and short-term instantaneous motion estimation for robust tracking. Evaluations on two challenging SatSOT and SV248S benchmarks confirm that SiamGM outperforms not only recent Siamese-based satellite trackers but also state-of-the-art Transformer trackers in both precision and success metrics. Notably, this highly unified architecture introduces only minor computational overhead, enabling real-time tracking at 130 frames per second (FPS).
Wuhan University（武汉大学）；The University of Sydney（悉尼大学）
Object tracking is a hot topic in computer vision. Thanks to the booming of the very high resolution (VHR) remote sensing techniques, it is now possible to track targets of interests in satellite videos. However, since the targets in the satellite videos are usually too small compared with the entire image, and too similar with the background, most state-of-the-art algorithms failed to track the target in satellite videos with a satisfactory accuracy. Due to the fact that optical flow shows the great potential to detect even the slight movement of the targets, we proposed a multi-frame optical…
University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Space Utilization, Chinese Academy of Sciences（中国科学院空间利用重点实验室）
The trajectory and boundary of an orbiting satellite are fundamental information for on-orbit repairing and manipulation by space robots. This task, however, is challenging owing to the freely and rapidly motion of on-orbiting satellites, the quickly varying background and the sudden change in illumination conditions. Traditional tracking usually relies on a single bounding box of the target object, however, more detailed information should be provided by visual tracking such as binary mask. In this paper, we proposed a SSLT (Saliency-based Semi-supervised Learning for Tracking) algorithm that…
SatSAM2: Motion-Constrained Video Object Tracking in Satellite Imagery using Promptable SAM2 and Kalman Priors
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Geospatial Engineering and Science, Sun Yat-sen University（中山大学测绘工程与科学学院）
Transformer-based visual object tracking has been utilized extensively. However, the Transformer structure is lack of enough inductive bias. In addition, only focusing on encoding the global feature does harm to modeling local details, which restricts the capability of tracking in aerial robots. Specifically, with local-modeling to global-search mechanism, the proposed tracker replaces the global encoder by a novel local-recognition encoder. In the employed encoder, a local-recognition attention and a local element correction network are carefully designed for reducing the global redundant inf…
Robustifying 3D Perception via Least-Squares Graphs for Multi-Agent Object Tracking
Industrial Systems Institute, Athena Research Center, Patras Science Park, Greece（希腊雅典研究中心工业系统研究所）；AviSense.AI, Patras Science Park, Greece（希腊帕特拉斯科学公园AviSense.AI）；Dpt. of Informatics & Telecom., University of Ioannina, Arta, Greece（希腊伊奥安妮大学信息与电信系）
In this paper, an autonomous method of satellite detection and tracking in images is implemented using optical flow. Optical flow is used to estimate the image velocities of detected objects in a series of space images. Given that most objects in an image will be stars, the overall image velocity from star motion is used to estimate the image's frame-to-frame motion. Objects seen to be moving with velocity profiles distinct from the overall image velocity are then classified as potential resident space objects. The detection algorithm is exercised using both simulated star images and ground-ba…
FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
Department of Electrical and Computer Engineering, University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校电子与计算机工程系）；Toyota Motor North America R&D（丰田北美研发）
DeTracker: Motion-decoupled Vehicle Detection and Tracking in Unstabilized Satellite Videos
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）
NASA Ames Research Center（美国国家航空航天局艾姆斯研究中心）；Bay Area Environmental Research Institute（湾区环境研究所）
Applications of satellite data in areas such as weather tracking and modeling, ecosystem monitoring, wildfire detection, and land-cover change are heavily dependent on the trade-offs to spatial, spectral and temporal resolutions of observations. In weather tracking, high-frequency temporal observations are critical and used to improve forecasts, study severe events, and extract atmospheric motion, among others. However, while the current generation of geostationary satellites have hemispheric coverage at 10-15 minute intervals, higher temporal frequency observations are ideal for studying meso…
Development and evaluation of automated localisation and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3D multi-object tracking
CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception
Institute for AI Industry Research, Tsinghua University（人工智能产业研究所，清华大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；School of Vehicle and Mobility, Tsinghua University（车辆与移动技术学院，清华大学）；Baidu Inc.（百度公司）
Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for t…
We present ongoing work to harness biological approaches to achieving highly efficient egocentric perception by combining the space-variant imaging architecture of the mammalian retina with Deep Learning methods. By pre-processing images collected by means of eye-tracking glasses to control the fixation locations of a software retina model, we demonstrate that we can reduce the input to a DCNN by a factor of 3, reduce the required number of training epochs and obtain over 98% classification rates when training and validating the system on a database of over 26,000 images of 9 object classes.
National University of Defense Technology（国防科技大学）；University of Oxford（牛津大学）；Sun Yat-sen University（中山大学）
Satellite video cameras can provide continuous observation for a large-scale area, which is important for many remote sensing applications. However, achieving moving object detection and tracking in satellite videos remains challenging due to the insufficient appearance information of objects and lack of high-quality datasets. In this paper, we first build a large-scale satellite video dataset with rich annotations for the task of moving object detection and tracking. This dataset is collected by the Jilin-1 satellite constellation and composed of 47 high-quality videos with 1,646,038 instance…
Key Laboratory of Machine Perception, Peking University, Shenzhen Graduate School（北京大学深圳研究生院机器感知重点实验室）；Computer Vision Lab, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）
Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map …
Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation
Optoelectronics Center, Viettel Aerospace Institute, Viettel Group（Viettel集团光学电子中心、 Viettel航空航天研究所）；University of Engineering and Technology, Vietnam National University（越南国家大学工程大学）；School of Electrical and Electronic Engineering, Hanoi University of Science and Technology（河内科学技术大学电子与电气工程学院）
Element AI；McGill University（麦吉尔大学）；Oxford / FDL（牛津大学 / FDL）；Global Witness（全球见证）
Cattle farming is responsible for 8.8\% of greenhouse gas emissions worldwide. In addition to the methane emitted due to their digestive process, the growing need for grazing areas is an important driver of deforestation. While some regulations are in place for preserving the Amazon against deforestation, these are being flouted in various ways, hence the need to scale and automate the monitoring of cattle ranching activities. Through a partnership with \textit{Global Witness}, we explore the feasibility of tracking and counting cattle at the continental scale from satellite imagery. With a li…
Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting dow…
Indian Institute of Technology, Roorkee（印度理工学院鲁尔基分校）
The presented study is an eye tracking experiment for high-resolution satellite (HRS) images. The reported experiment explores the Area Of Interest (AOI) based analysis of eye fixation data for complex HRS images. The study reflects the requisite of reference data for bottom-up saliency-based segmentation and the struggle of eye tracking data analysis for complex satellite images. The intended fixation data analysis aims towards the reference data creation for bottom-up saliency-based segmentation of high-resolution satellite images. The analytical outcome of this experimental study provides a…
This paper introduces a novel perception framework that has the ability to identify and track objects in autonomous vehicle's field of view. The proposed algorithms don't require any training for achieving this goal. The framework makes use of ego-vehicle's pose estimation and a KD-Tree-based segmentation algorithm to generate object clusters. In turn, using a VFH technique, the geometry of each identified object cluster is translated into a multi-modal PDF and a motion model is initiated with every new object cluster for the purpose of robust spatio-temporal tracking. The methodology further …
University of Tübingen（图宾根大学）；Technical University of Munich（慕尼黑工业大学）；Leibniz-Institut für Wissensmedien（莱布尼茨知识媒体研究所）
The ability for a teacher to engage all students in active learning processes in classroom constitutes a crucial prerequisite for enhancing students achievement. Teachers' attentional processes provide important insights into teachers' ability to focus their attention on relevant information in the complexity of classroom interaction and distribute their attention across students in order to recognize the relevant needs for learning. In this context, mobile eye tracking is an innovative approach within teaching effectiveness research to capture teachers' attentional processes while teaching. H…
Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor bounding the performance. Motivated by this observation, we propose a multi-level fusion method that combines different representations (RGB and pseudo-LiDAR) and temporal information across multiple frames…
Uber Advanced Technologies Group（Uber先进技术集团）；University of Toronto（多伦多大学）
We tackle the problem of joint perception and motion forecasting in the context of self-driving vehicles. Towards this goal we propose PnPNet, an end-to-end model that takes as input sequential sensor data, and outputs at each time step object tracks and their future trajectories. The key component is a novel tracking module that generates object tracks online from detections and exploits trajectory level features for motion forecasting. Specifically, the object tracks get updated at each time step by solving both the data association problem and the trajectory estimation problem. Importantly,…
University of California at Riverside（加州大学河滨分校）；Toyota Motor North America（丰田汽车北美公司）
Object detection plays a fundamental role in enabling Cooperative Driving Automation (CDA), which is regarded as the revolutionary solution to addressing safety, mobility, and sustainability issues of contemporary transportation systems. Although current computer vision technologies could provide satisfactory object detection results in occlusion-free scenarios, the perception performance of onboard sensors could be inevitably limited by the range and occlusion. Owing to flexible position and pose for sensor installation, infrastructure-based detection and tracking systems can enhance the perc…
Cloud motion winds (CMW) are routinely derived by tracking features in sequential geostationary satellite infrared cloud imagery. In this paper, we explore the cloud motion winds algorithm based on data-driven deep learning approach, and different from conventional hand-craft feature tracking and correlation matching algorithms, we use deep learning model to automatically learn the motion feature representations and directly output the field of cloud motion winds. In addition, we propose a novel large-scale cloud motion winds dataset (CMWD) for training deep learning models. We also try to use…
InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarios
School of Intelligent Systems Engineering, Sun Yat-sen University（智能系统工程学院，中山大学）；Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai)（广东海洋科学与工程实验室（珠海））
This paper describes an unsupervised machine learning methodology capable of target tracking and background suppression via a novel dual-model approach. ``Jekyll`` produces a video bit-mask describing an estimate of the locations of moving objects, and ``Hyde`` outputs a pseudo-background frame to subtract from the original input image sequence. These models were trained with a custom-modified version of Cross Entropy Loss. Simulated data were used to compare the performance of Jekyll and Hyde against a more traditional supervised Machine Learning approach. The results from these comparisons s…
Event-RGB Adaptive Tracking for Nighttime Highway Perception
School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）；School of Intelligence Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学智能科学与工程学院）
Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；State Key Laboratory of Industrial Control Technology, Zhejiang University（浙江大学工业控制技术国家重点实验室）；Hangzhou Dianzi University（杭州电子科技大学）；Pillar of Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计 pillar）
Indian Institute of Technology Ropar（印度理工学院罗巴尔分校）；Monash University（莫纳什大学）
We present \textbf{FakeET}-- an eye-tracking database to understand human visual perception of \emph{deepfake} videos. Given that the principal purpose of deepfakes is to deceive human observers, FakeET is designed to understand and evaluate the ease with which viewers can detect synthetic video artifacts. FakeET contains viewing patterns compiled from 40 users via the \emph{Tobii} desktop eye-tracker for 811 videos from the \textit{Google Deepfake} dataset, with a minimum of two viewings per video. Additionally, EEG responses acquired via the \emph{Emotiv} sensor are also available. The compi…
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Video
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）
AquaFeat+: an Underwater Vision Learning-based Enhancement Method for Object Detection, Classification, and Tracking
Centro de Ciências Computacionais (C3), Universidade Federal do Rio Grande (FURG), Brazil（巴西里约格兰德联邦大学计算科学研究中心（C3））；Instituto de Ciências Exatas e Tecnologia (ICET), Universidade Federal do Amazonas (UFAM), Brazil（巴西亚马逊联邦大学精确科学与技术研究所（ICET））
