How to combine the complementary capabilities of an ensemble of different algorithms has been of central interest in visual object tracking. A significant progress on such a problem has been achieved, but considering short-term tracking scenarios. Instead, long-term tracking settings have been substantially ignored by the solutions. In this paper, we explicitly consider long-term tracking scenarios and provide a framework, named CoCoLoT, that combines the characteristics of complementary visual trackers to achieve enhanced long-term tracking performance. CoCoLoT perceives whether the trackers are following the target object through an online learned deep verification model, and accordingly activates a decision policy which selects the best performing tracker as well as it corrects the performance of the failing one. The proposed methodology is evaluated extensively and the comparison with several other solutions reveals that it competes favourably with the state-of-the-art on the most popular long-term visual tracking benchmarks.
Faculty of Computer and Information Science, University of Ljubljana（卢布尔雅那大学计算机与信息科学学院）；Faculty of Electrical Engineering, Czech Technical University in Prague（布拉格捷克理工大学电气工程学院）
Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker - D3S, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid objec…
The Center for Machine Perception, FEE CTU in Prague（捷克布拉格技术大学电气工程学院机器感知中心）；Faculty of Civil Engineering, CTU in Prague（捷克布拉格技术大学土木工程学院）
In this paper, we propose a novel method for visual object tracking called HMMTxD. The method fuses observations from complementary out-of-the box trackers and a detector by utilizing a hidden Markov model whose latent states correspond to a binary vector expressing the failure of individual trackers. The Markov model is trained in an unsupervised way, relying on an online learned detector to provide a source of tracker-independent information for a modified Baum- Welch algorithm that updates the model w.r.t. the partially annotated data. We show the effectiveness of the proposed method on com…
Eskisehir Technical University（埃斯基谢希尔技术大学）；Eskisehir Osmangazi University（埃斯基谢希尔奥斯曼加齐大学）；Technical University of Munich（慕尼黑工业大学）
Robust object tracking requires knowledge of tracked objects' appearance, motion and their evolution over time. Although motion provides distinctive and complementary information especially for fast moving objects, most of the recent tracking architectures primarily focus on the objects' appearance information. In this paper, we propose a two-stream deep neural network tracker that uses both spatial and temporal features. Our architecture is developed over ATOM tracker and contains two backbones: (i) 2D-CNN network to capture appearance features and (ii) 3D-CNN network to capture motion featur…
University of Ljubljana（卢布尔雅那大学）；Czech Technical University in Prague（布拉格捷克理工大学）
Template-based discriminative trackers are currently the dominant tracking paradigm due to their robustness, but are restricted to bounding box tracking and a limited range of transformation models, which reduces their localization accuracy. We propose a discriminative single-shot segmentation tracker -- D3S2, which narrows the gap between visual object tracking and video object segmentation. A single-shot network applies two target models with complementary geometric properties, one invariant to a broad range of transformations, including non-rigid deformations, the other assuming a rigid obj…
Faculty of Computer and Information Science, University of Ljubljana（卢布尔雅那大学计算机与信息科学学院）；Centre for Computational Neuroscience and Cognitive Robotics, University of Birmingham（伯明翰大学计算神经科学与认知机器人中心）
The problem of visual tracking evaluation is sporting a large variety of performance measures, and largely suffers from lack of consensus about which measures should be used in experiments. This makes the cross-paper tracker comparison difficult. Furthermore, as some measures may be less effective than others, the tracking results may be skewed or biased towards particular tracking aspects. In this paper we revisit the popular performance measures and tracker performance visualizations and analyze them theoretically and experimentally. We show that several measures are equivalent from the poin…
Harbin Institute of Technology（哈尔滨工业大学）；Shenzhen Graduate School, Harbin Institute of Technology（哈尔滨工业大学深圳研究生院）
Discriminative Correlation Filters based tracking algorithms exploiting conventional handcrafted features have achieved impressive results both in terms of accuracy and robustness. Template handcrafted features have shown excellent performance, but they perform poorly when the appearance of target changes rapidly such as fast motions and fast deformations. In contrast, statistical handcrafted features are insensitive to fast states changes, but they yield inferior performance in the scenarios of illumination variations and background clutters. In this work, to achieve an efficient tracking per…
Visual object tracking with RGB and thermal infrared (TIR) spectra available, shorted in RGBT tracking, is a novel and challenging research topic which draws increasing attention nowadays. In this paper, we propose an RGBT tracker which takes spatio-temporal clues into account for robust appearance model learning, and simultaneously, constructs an adaptive fusion sub-network for cross-modal interactions. Unlike most existing RGBT trackers that implement object tracking tasks with only spatial information included, temporal information is further considered in this method. Specifically, differe…
Object-to-camera motion produces a variety of apparent motion patterns that significantly affect performance of short-term visual trackers. Despite being crucial for designing robust trackers, their influence is poorly explored in standard benchmarks due to weakly defined, biased and overlapping attribute annotations. In this paper we propose to go beyond pre-recorded benchmarks with post-hoc annotations by presenting an approach that utilizes omnidirectional videos to generate realistic, consistently annotated, short-term tracking scenarios with exactly parameterized motion patterns. We have …
Correlation Filter-based trackers have recently achieved excellent performance, showing great robustness to challenging situations exhibiting motion blur and illumination changes. However, since the model that they learn depends strongly on the spatial layout of the tracked object, they are notoriously sensitive to deformation. Models based on colour statistics have complementary traits: they cope well with variation in shape, but suffer when illumination is not consistent throughout a sequence. Moreover, colour distributions alone can be insufficiently discriminative. In this paper, we show t…
In recent years, the multiple-stage strategy has become a popular trend for visual tracking. This strategy first utilizes a base tracker to coarsely locate the target and then exploits a refinement module to obtain more accurate results. However, existing refinement modules suffer from the limited transferability and precision. In this work, we propose a novel, flexible and accurate refinement module called Alpha-Refine, which exploits a precise pixel-wise correlation layer together with a spatial-aware non-local layer to fuse features and can predict three complementary outputs: bounding box,…
Linköping University（林雪平大学）；Department of Electrical Engineering（电气工程系）；CVL（CVL（这里CVL可能是缩写，按原文保留））；Sweden（瑞典（注意：根据规则，Sweden是国家，不是机构，所以应该只提取大学级别。这里原文本中Linköping University是大学，是机构；Department of Electrical Engineering是系，不是独立机构级别，所以应提取Linköping University。）}]}修正后：{）
In the field of generic object tracking numerous attempts have been made to exploit deep features. Despite all expectations, deep trackers are yet to reach an outstanding level of performance compared to methods solely based on handcrafted features. In this paper, we investigate this key issue and propose an approach to unlock the true potential of deep features for tracking. We systematically study the characteristics of both deep and shallow features, and their relation to tracking accuracy and robustness. We identify the limited data and low spatial resolution as the main challenges, and pr…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Siamese tracking has achieved groundbreaking performance in recent years, where the essence is the efficient matching operator cross-correlation and its variants. Besides the remarkable success, it is important to note that the heuristic matching network design relies heavily on expert experience. Moreover, we experimentally find that one sole matching operator is difficult to guarantee stable tracking in all challenging environments. Thus, in this work, we introduce six novel matching operators from the perspective of feature fusion instead of explicit similarity learning, namely Concatenatio…
Shenzhen Graduate School, Harbin Institute of Technology（哈尔滨工业大学深圳研究生院）；Graduate School at Shenzhen, Tsinghua University（清华大学深圳研究生院）
Correlation filter (CF) based tracking algorithms have demonstrated favorable performance recently. Nevertheless, the top performance trackers always employ complicated optimization methods which constraint their real-time applications. How to accelerate the tracking speed while retaining the tracking accuracy is a significant issue. In this paper, we propose a multi-level CF-based tracking approach named MLCFT which further explores the potential capacity of CF with two-stage detection: primal detection and oriented re-detection. The cascaded detection scheme is simple but competent to preven…
Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Beijing Institute of Technology（北京理工大学）；School of Electronic and Information Engineering, Anhui University（安徽大学电子与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Peking University（北京大学计算机科学学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；School of Information Science and Engineering（信息科学与工程学院）；School of Mathematics（数学学院）；Purple Mountain Laboratories（紫金山实验室）
RGB-Thermal object tracking attempt to locate target object using complementary visual and thermal infrared data. Existing RGB-T trackers fuse different modalities by robust feature representation learning or adaptive modal weighting. However, how to integrate dual attention mechanism for visual tracking is still a subject that has not been studied yet. In this paper, we propose two visual attention mechanisms for robust RGB-T object tracking. Specifically, the local attention is implemented by exploiting the common visual attention of RGB and thermal data to train deep classifiers. We also in…
With the guaranteed discrimination and efficiency of spatial appearance model, Discriminative Correlation Filters (DCF-) based tracking methods have achieved outstanding performance recently. However, the construction of effective temporal appearance model is still challenging on account of filter degeneration becomes a significant factor that causes tracking failures in the DCF framework. To encourage temporal continuity and to explore the smooth variation of target appearance, we propose to enhance low-rank structure of the learned filters, which can be realized by constraining the successiv…
RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. However, how to effectively represent RGBT data for visual tracking remains unstudied well. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target represent…
The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for ef…
Existing Visual Object Tracking (VOT) only takes the target area in the first frame as a template. This causes tracking to inevitably fail in fast-changing and crowded scenes, as it cannot account for changes in object appearance between frames. To this end, we revamped the tracking framework with Progressive Context Encoding Transformer Tracker (ProContEXT), which coherently exploits spatial and temporal contexts to predict object motion trajectories. Specifically, ProContEXT leverages a context-aware self-attention module to encode the spatial and temporal context, refining and updating the …
SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间智能信息处理国家级重点实验室）；Beijing Institute of Technology, Zhuhai, Guangdong（珠海，广东，北京理工大学）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德克萨斯大学）；Department of Electrical and Computer Engineering, Mississippi State University（电气与计算机工程系，密苏里州立大学）
UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Peking University（北京大学计算机学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
Object proposal generation methods have been widely applied to many computer vision tasks. However, existing object proposal generation methods often suffer from the problems of motion blur, low contrast, deformation, etc., when they are applied to video related tasks. In this paper, we propose an effective and highly accurate target-specific object proposal generation (TOPG) method, which takes full advantage of the context information of a video to alleviate these problems. Specifically, we propose to generate target-specific object proposals by integrating the information of two important o…
This paper presents a novel hybrid representation learning framework for streaming data, where an image frame in a video is modeled by an ensemble of two distinct deep neural networks; one is a low-bit quantized network and the other is a lightweight full-precision network. The former learns coarse primary information with low cost while the latter conveys residual information for high fidelity to original representations. The proposed parallel architecture is effective to maintain complementary information since fixed-point arithmetic can be utilized in the quantized network and the lightweig…
This paper investigates how to perform robust visual tracking in adverse and challenging conditions using complementary visual and thermal infrared data (RGBT tracking). We propose a novel deep network architecture called qualityaware Feature Aggregation Network (FANet) for robust RGBT tracking. Unlike existing RGBT trackers, our FANet aggregates hierarchical deep features within each modality to handle the challenge of significant appearance changes caused by deformation, low illumination, background clutter and occlusion. In particular, we employ the operations of max pooling to transform th…
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
China University of Mining and Technology（中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education（矿山数字化教育部工程研究中心）；University of Auckland（奥克兰大学）
Jiangnan University（江南大学）；University of Surrey（萨里大学）
We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Feature-level fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the sup…
The tracking-by-detection paradigm today has become the dominant method for multi-object tracking and works by detecting objects in each frame and then performing data association across frames. However, its sequential frame-wise matching property fundamentally suffers from the intermediate interruptions in a video, such as object occlusions, fast camera movements, and abrupt light changes. Moreover, it typically overlooks temporal information beyond the two frames for matching. In this paper, we investigate an alternative by treating object association as clip-wise matching. Our new perspecti…
A robust and fast automatic moving object detection and tracking system is essential to characterize target object and extract spatial and temporal information for different functionalities including video surveillance systems, urban traffic monitoring and navigation, robotic. In this dissertation, I present a collaborative Spatial Pyramid Context-aware moving object detection and Tracking system. The proposed visual tracker is composed of one master tracker that usually relies on visual object features and two auxiliary trackers based on object temporal motion information that will be called …
Baidu Inc.（百度公司）；Huazhong University of Science and Technology（华中科技大学）；The University of Hong Kong（香港大学）
Multi-object tracking (MOT) aims at estimating bounding boxes and identities of objects across video frames. Detection boxes serve as the basis of both 2D and 3D MOT. The inevitable changing of detection scores leads to object missing after tracking. We propose a hierarchical data association strategy to mine the true objects in low-score detection boxes, which alleviates the problems of object missing and fragmented trajectories. The simple and generic data association strategy shows effectiveness under both 2D and 3D settings. In 3D scenarios, it is much easier for the tracker to predict obj…
Anhui University（安徽大学）；Texas State University（得克萨斯州立大学）
RGB-Thermal (RGB-T) object tracking receives more and more attention due to the strongly complementary benefits of thermal information to visible data. However, RGB-T research is limited by lacking a comprehensive evaluation platform. In this paper, we propose a large-scale video benchmark dataset for RGB-T tracking.It has three major advantages over existing ones: 1) Its size is sufficiently large for large-scale performance evaluation (total frame number: 234K, maximum frame per sequence: 8K). 2) The alignment between RGB-T sequence pairs is highly accurate, which does not need pre- or post-…
SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking
College of Computer Science and Engineering, Guilin University of Technology（桂林理工大学计算机科学与工程学院）；School of Information Engineering, Wuhan University of Technology（武汉理工大学信息工程学院）
FEMOT: Multi-Object Tracking using Frame and Event Cameras
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理全国重点实验室）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University（广西多源信息挖掘与安全重点实验室，广西师范大学）；University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University（教育智能技术大学工程研究中心，广西师范大学）；Guangxi Key Laboratory of Machine Vision and Intelligent Control, Wuzhou University（广西机器视觉与智能控制重点实验室，梧州大学）
OST: Efficient One-stream Network for 3D Single Object Tracking in Point Clouds
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Economics and Management, China University of Mining and Technology（中国矿业大学经济管理学院）；School of Software, Tsinghua University（清华大学软件学院）；Dalian University of Technology（大连理工大学）
The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）
This paper investigates two typical image-type representations for event camera-based tracking: time surface (TS) and event map (EM). Based on the original TS-based tracker, we make use of these two representations' complementary strengths to develop an enhanced version. The proposed tracker consists of a general strategy to evaluate the optimization problem's degeneracy online and then switch proper representations. Both TS and EM are motion- and scene-dependent, and thus it is important to figure out their limitations in tracking. We develop six tracker variations and conduct a thorough comp…
In this paper we address the problem of tracking multiple speakers via the fusion of visual and auditory information. We propose to exploit the complementary nature of these two modalities in order to accurately estimate smooth trajectories of the tracked persons, to deal with the partial or total absence of one of the modalities over short periods of time, and to estimate the acoustic status -- either speaking or silent -- of each tracked person along time. We propose to cast the problem at hand into a generative audio-visual fusion (or association) model formulated as a latent-variable tempo…
SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker
School of Artificial Intelligence (School of Software), Yanshan University（燕山大学人工智能学院（软件学院））；School of Software, Beihang University（北航软件学院）；Hebei Key Laboratory of Computer Virtual Technology and System Integration（河北省计算机虚拟技术与系统集成重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（多模态人工智能系统国家重点实验室（MAIS），CASIA）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院自动化实验室）
Shanghai Jiao Tong University（上海交通大学）；CloudWalk Technology Co., Ltd（云从科技有限公司）；Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；JD Explore Academy in JD.com（京东集团京东探索研究院）；University of Sydney（悉尼大学）
Unmanned aerial vehicle (UAV) tracking is of great significance for a wide range of applications, such as delivery and agriculture. Previous benchmarks in this area mainly focused on small-scale tracking problems while ignoring the amounts of data, types of data modalities, diversities of target categories and scenarios, and evaluation protocols involved, greatly hiding the massive power of deep UAV tracking. In this work, we propose WebUAV-3M, the largest public UAV tracking benchmark to date, to facilitate both the development and evaluation of deep UAV trackers. WebUAV-3M contains over 3.3 …
