Visual object tracking with the visible (RGB) and thermal infrared (TIR) electromagnetic waves, shorted in RGBT tracking, recently draws increasing attention in the tracking community. Considering the rapid development of deep learning, a survey for the recent deep neural network based RGBT trackers is presented in this paper. Firstly, we give brief introduction for the RGBT trackers concluded into this category. Then, a comparison among the existing RGBT trackers on several challenging benchmarks is given statistically. Specifically, MDNet and Siamese architectures are the two mainstream frameworks in the RGBT community, especially the former. Trackers based on MDNet achieve higher performance while Siamese-based trackers satisfy the real-time requirement. In summary, since the large-scale dataset LasHeR is published, the integration of end-to-end framework, e.g., Siamese and Transformer, should be further considered to fulfil the real-time as well as more robust performance. Furthermore, the mathematical meaning should be more considered during designing the network. This survey can be treated as a look-up-table for researchers who are concerned about RGBT tracking.
Visual object tracking with RGB and thermal infrared (TIR) spectra available, shorted in RGBT tracking, is a novel and challenging research topic which draws increasing attention nowadays. In this paper, we propose an RGBT tracker which takes spatio-temporal clues into account for robust appearance model learning, and simultaneously, constructs an adaptive fusion sub-network for cross-modal interactions. Unlike most existing RGBT trackers that implement object tracking tasks with only spatial information included, temporal information is further considered in this method. Specifically, differe…
The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for ef…
RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. However, how to effectively represent RGBT data for visual tracking remains unstudied well. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target represent…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Feature-level fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the sup…
This paper investigates how to perform robust visual tracking in adverse and challenging conditions using complementary visual and thermal infrared data (RGBT tracking). We propose a novel deep network architecture called qualityaware Feature Aggregation Network (FANet) for robust RGBT tracking. Unlike existing RGBT trackers, our FANet aggregates hierarchical deep features within each modality to handle the challenge of significant appearance changes caused by deformation, low illumination, background clutter and occlusion. In particular, we employ the operations of max pooling to transform th…
Visual object tracking, as a fundamental task in computer vision, has drawn much attention in recent years. To extend trackers to a wider range of applications, researchers have introduced information from multiple modalities to handle specific scenes, which is a promising research prospect with emerging methods and benchmarks. To provide a thorough review of multi-modal track-ing, we summarize the multi-modal tracking algorithms, especially visible-depth (RGB-D) tracking and visible-thermal (RGB-T) tracking in a unified taxonomy from different aspects. Second, we provide a detailed descriptio…
RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a Large-scale High-diversity benchmark for RGBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and…
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University（广西多源信息挖掘与安全重点实验室，广西师范大学）；University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University（教育智能技术大学工程研究中心，广西师范大学）；Guangxi Key Laboratory of Machine Vision and Intelligent Control, Wuzhou University（广西机器视觉与智能控制重点实验室，梧州大学）
How to perform effective information fusion of different modalities is a core factor in boosting the performance of RGBT tracking. This paper presents a novel deep fusion algorithm based on the representations from an end-to-end trained convolutional neural network. To deploy the complementarity of features of all layers, we propose a recursive strategy to densely aggregate these features that yield robust representations of target objects in each modality. In different modalities, we propose to prune the densely aggregated features of all modalities in a collaborative way. In a specific, we e…
DRGBT-1K: A Large-scale High-quality Benchmark for Dynamic RGBT Tracking
Anhui University（安徽大学）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电子信息采集与保护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
Anhui University（安徽大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Low-quality modalities contain not only a lot of noisy information but also some discriminative features in RGBT tracking. However, the potentials of low-quality modalities are not well explored in existing RGBT tracking algorithms. In this work, we propose a novel duality-gated mutual condition network to fully exploit the discriminative information of all modalities while suppressing the effects of data noise. In specific, we design a mutual condition module, which takes the discriminative information of a modality as the condition to guide feature learning of target appearance in another mo…
East China University of Science and Technology（华东理工大学）
For both visible and infrared images have their own advantages and disadvantages, RGBT tracking has attracted more and more attention. The key points of RGBT tracking lie in feature extraction and feature fusion of visible and infrared images. Current RGBT tracking methods mostly pay attention to both individual features (features extracted from images of a single camera) and common features (features extracted and fused from an RGB camera and a thermal camera), while pay less attention to the different and dynamic contributions of individual features and common features for different sequence…
AFter: Attention-based Fusion Router for RGBT Tracking
Anhui University（安徽大学）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）
Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey大学视觉、语音与信号处理中心）
Classifying the confusing samples in the course of RGBT tracking is a quite challenging problem, which hasn't got satisfied solution. Existing methods only focus on enlarging the boundary between positive and negative samples, however, the structured information of samples might be harmed, e.g., confusing positive samples are closer to the anchor than normal positive samples.To handle this problem, we propose a novel Multi-Modal Multi-Margin Metric Learning framework, named M$^5$L for RGBT tracking in this paper. In particular, we design a multi-margin structured loss to distinguish the confus…
RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation
College of Command and Control Engineering, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；School of Future Technology, Dalian University of Technology（大连理工大学未来技术学院）；School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）
Cross-modulated Attention Transformer for RGBT Tracking
School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；iFLYTEK CO.LTD.（科大讯飞股份有限公司）
RGB and thermal source data suffer from both shared and specific challenges, and how to explore and exploit them plays a critical role to represent the target appearance in RGBT tracking. In this paper, we propose a novel challenge-aware neural network to handle the modality-shared challenges (e.g., fast motion, scale variation and occlusion) and the modality-specific ones (e.g., illumination variation and thermal crossover) for RGBT tracking. In particular, we design several parameter-shared branches in each layer to model the target appearance under the modality-shared challenges, and severa…
Dynamic Disentangled Fusion Network for RGBT Tracking
Anhui University（安徽大学）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Peking University（北京大学计算机学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
Over these years, Correlation Filter-based Trackers (CFTs) have aroused increasing interests in the field of visual object tracking, and have achieved extremely compelling results in different competitions and benchmarks. In this paper, our goal is to review the developments of CFTs with extensive experimental results. 11 trackers are surveyed in our work, based on which a general framework is summarized. Furthermore, we investigate different training schemes for correlation filters, and also discuss various effective improvements that have been made recently. Comprehensive experiments have be…
Khalifa University of Science and Technology（哈利法科技大学）；ETH Zürich（苏黎世联邦理工学院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Linköping University（林雪平大学）；Czech Technical University（捷克理工大学）
Accurate and robust visual object tracking is one of the most challenging and fundamental computer vision problems. It entails estimating the trajectory of the target in an image sequence, given only its initial location, and segmentation, or its rough approximation in the form of a bounding box. Discriminative Correlation Filters (DCFs) and deep Siamese Networks (SNs) have emerged as dominating tracking paradigms, which have led to significant progress. Following the rapid evolution of visual object tracking in the last decade, this survey presents a systematic and thorough review of more tha…
Tianjin University（天津大学）；Key Research Center for Surface Monitoring and Analysis of Cultural Relics（文物本体表面监测与分析研究重点科研基地）
Visual object tracking is an important task in computer vision, which has many real-world applications, e.g., video surveillance, visual navigation. Visual object tracking also has many challenges, e.g., object occlusion and deformation. To solve above problems and track the target accurately and efficiently, many tracking algorithms have emerged in recent years. This paper presents the rationale and representative works of two most popular tracking frameworks in past ten years, i.e., the corelation filter and Siamese network for object tracking. Then we present some deep learning based tracki…
The University of Adelaide（阿德莱德大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；State University of New York, Binghamton（纽约州立大学宾汉姆顿分校）
Visual object tracking is a significant computer vision task which can be applied to many domains such as visual surveillance, human computer interaction, and video compression. In the literature, researchers have proposed a variety of 2D appearance models. To help readers swiftly learn the recent advances in 2D appearance models for visual object tracking, we contribute this survey, which provides a detailed review of the existing 2D appearance models. In particular, this survey takes a module-based architecture that enables readers to easily grasp the key points of visual object tracking. In…
UASTrack: A Unified Adaptive Selection Framework with Modality-Customization in Single Object Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音和信号处理中心）
Cooperative Medianet Innovation Center（合作 medianet 创新中心）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；CloudWalk Technology Co., Ltd（云从科技股份有限公司）；Shanghai AI Laboratory（上海人工智能实验室）
China University of Mining and Technology（中国矿业大学）；Nanyang Technological University（南洋理工大学）
Object segmentation and object tracking are fundamental research area in the computer vision community. These two topics are diffcult to handle some common challenges, such as occlusion, deformation, motion blur, and scale variation. The former contains heterogeneous object, interacting object, edge ambiguity, and shape complexity. And the latter suffers from difficulties in handling fast motion, out-of-view, and real-time processing. Combining the two problems of video object segmentation and tracking (VOST) can overcome their respective difficulties and improve their performance. VOST can be…
Malek Ashtar University of Technology（马利克·阿什塔尔理工大学）
Object tracking is one of the foremost assignments in computer vision that has numerous commonsense applications such as traffic monitoring, robotics, autonomous vehicle tracking, and so on. Different researches have been tried later a long time, but since of diverse challenges such as occlusion, illumination variations, fast motion, etc. researches in this area continues. In this paper, different strategies of the following objects are inspected and a comprehensive classification is displayed that classified the following strategies into four fundamental categories of feature-based, segmentat…
Omni Survey for Multimodality Analysis in Visual Object Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；School of Computer Science and Technology, China University of Mining and Technology（中国矿业大学计算机科学与技术学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey 大学视觉、语音与信号处理中心）
Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey
Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院，浙江大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Electrical and Computer Engineering, the University of Washington（华盛顿大学电气与计算机工程系）
Data association is a key step within the multi-object tracking pipeline that is notoriously challenging due to its combinatorial nature. A popular and general way to formulate data association is as the NP-hard multidimensional assignment problem (MDAP). Over the last few years, data-driven approaches to assignment have become increasingly prevalent as these techniques have started to mature. We focus this survey solely on learning algorithms for the assignment step of multi-object tracking, and we attempt to unify various methods by highlighting their connections to linear assignment as well…
This paper presents a review of human activity recognition and behaviour understanding in video sequence. The key objective of this paper is to provide a general review on the overall process of a surveillance system used in the current trend. Visual surveillance system is directed on automatic identification of events of interest, especially on tracking and classification of moving objects. The processing step of the video surveillance system includes the following stages: Surrounding model, object representation, object tracking, activity recognition and behaviour understanding. It describes…
Kyungpook National University（庆北国立大学）；Qatar University（卡塔尔大学）
Visual object tracking is an important computer vision problem with numerous real-world applications including human-computer interaction, autonomous vehicles, robotics, motion-based recognition, video indexing, surveillance and security. In this paper, we aim to extensively review the latest trends and advances in the tracking algorithms and evaluate the robustness of trackers in the presence of noise. The first part of this work comprises a comprehensive survey of recently proposed tracking algorithms. We broadly categorize trackers into correlation filter based trackers and the others as no…
Phase contrast microscopy (PCM) has been widely used in biomedicine research, which allows users to observe objectives without staining or killing them. One important related research is to employ PCM to monitor live cells. How to segment cell populations in obtained PCM images gains more and more attention as its a critical step for downstream applications, such as cell tracking, cell classification and others. Many papers have been published to deal with this problem from different perspectives. In this paper we aim to present a comprehensive review on the development of PCM cell population …
CHOTA: A Higher Order Accuracy Metric for Cell Tracking
Institute for Information Processing – Leibniz University Hannover（汉诺威莱布尼茨大学信息处理研究所）；IT4Innovations, VSB – Technical University of Ostrava（俄斯特拉发技术大学VSB IT4Innovations研究院）
Yazd University（亚兹德大学）；University of Alberta（阿尔伯塔大学）；Sharif University of Technology（谢里夫理工大学）
Visual target tracking is one of the most sought-after yet challenging research topics in computer vision. Given the ill-posed nature of the problem and its popularity in a broad range of real-world scenarios, a number of large-scale benchmark datasets have been established, on which considerable methods have been developed and demonstrated with significant progress in recent years -- predominantly by recent deep learning (DL)-based methods. This survey aims to systematically investigate the current DL-based visual tracking methods, benchmark datasets, and evaluation metrics. It also extensive…
Kyungpook National University（庆北国立大学）；Information Technology University（信息技术大学）；University of Warwick（华威大学）
In recent years visual object tracking has become a very active research area. An increasing number of tracking algorithms are being proposed each year. It is because tracking has wide applications in various real world problems such as human-computer interaction, autonomous vehicles, robotics, surveillance and security just to name a few. In the current study, we review latest trends and advances in the tracking area and evaluate the robustness of different trackers based on the feature extraction methods. The first part of this work comprises a comprehensive survey of the recently proposed t…
