In this paper, the main task we aim to tackle is the multi-instance semi-supervised video object segmentation across a sequence of frames where only the first-frame box-level ground-truth is provided. Detection-based algorithms are widely adopted to handle this task, and the challenges lie in the selection of the matching method to predict the result as well as to decide whether to update the target template using the newly predicted result. The existing methods, however, make these selections in a rough and inflexible way, compromising their performance. To overcome this limitation, we propose a novel approach which utilizes reinforcement learning to make these two decisions at the same time. Specifically, the reinforcement learning agent learns to decide whether to update the target template according to the quality of the predicted result. The choice of the matching method will be determined at the same time, based on the action history of the reinforcement learning agent. Experiments show that our method is almost 10 times faster than the previous state-of-the-art method with even higher accuracy (region similarity of 69.1% on DAVIS 2017 dataset).
This work presents a framework for tracking head movements and capturing the movements of the mouth and both the eyebrows in real-time. We present a head tracker which is a combination of a optical flow and a template based tracker. The estimation of the optical flow head tracker is used as starting point for the template tracker which fine-tunes the head estimation. This approach together with re-updating the optical flow points prevents the head tracker from drifting. This combination together with our switching scheme, makes our tracker very robust against fast movement and motion-blur. We …
Sparse representation has been widely studied in visual tracking, which has shown promising tracking performance. Despite a lot of progress, the visual tracking problem is still a challenging task due to appearance variations over time. In this paper, we propose a novel sparse tracking algorithm that well addresses temporal appearance changes, by enforcing template representability and temporal consistency (TRAC). By modeling temporal consistency, our algorithm addresses the issue of drifting away from a tracking target. By exploring the templates' long-term-short-term representability, the pr…
A robust algorithm solution is proposed for tracking an object in complex video scenes. In this solution, the bootstrap particle filter (PF) is initialized by an object detector, which models the time-evolving background of the video signal by an adaptive Gaussian mixture. The motion of the object is expressed by a Markov model, which defines the state transition prior. The color and texture features are used to represent the object, and a marginal likelihood based feature fusion approach is proposed. A corresponding object template model updating procedure is developed to account for possible…
Shanghai Jiao Tong University（上海交通大学）；Nanjing University of Science and Technology（南京理工大学）；University of Sydney（悉尼大学）
In this paper, we propose a novel matching based tracker by investigating the relationship between template matching and the recent popular correlation filter based trackers (CFTs). Compared to the correlation operation in CFTs, a sophisticated similarity metric termed "mutual buddies similarity" (MBS) is proposed to exploit the relationship of multiple reciprocal nearest neighbors for target matching. By doing so, our tracker obtains powerful discriminative ability on distinguishing target and background as demonstrated by both empirical and theoretical analyses. Besides, instead of utilizing…
Boosting performance of the offline trained siamese trackers is getting harder nowadays since the fixed information of the template cropped from the first frame has been almost thoroughly mined, but they are poorly capable of resisting target appearance changes. Existing trackers with template updating mechanisms rely on time-consuming numerical optimization and complex hand-designed strategies to achieve competitive performance, hindering them from real-time tracking and practical applications. In this paper, we propose a novel tracking framework built on top of a space-time memory network th…
Seoul National University（首尔大学）；Chung-Ang University（中央大学）
One of the major challenges of model-free visual tracking problem has been the difficulty originating from the unpredictable and drastic changes in the appearance of objects we target to track. Existing methods tackle this problem by updating the appearance model on-line in order to adapt to the changes in the appearance. Despite the success of these methods however, inaccurate and erroneous updates of the appearance model result in a tracker drift. In this paper, we introduce a novel real-time visual tracking algorithm based on a template selection strategy constructed by deep reinforcement l…
Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；ETH Zürich（苏黎世联邦理工学院）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）；Linköping University（林雪平大学）
Siamese approaches address the visual tracking problem by extracting an appearance template from the current frame, which is used to localize the target in the next frame. In general, this template is linearly combined with the accumulated template from the previous frame, resulting in an exponential decay of information over time. While such an approach to updating has led to improved results, its simplicity limits the potential gain likely to be obtained by learning to update. Therefore, we propose to replace the handcrafted update function with a method which learns to update. We use a conv…
Dalian University of Technology（大连理工大学）；The University of Sydney（悉尼大学）；China Science IntelliCloud Technology Co., Ltd（中科智云科技有限公司）
The fully-convolutional siamese network based on template matching has shown great potentials in visual tracking. During testing, the template is fixed with the initial target feature and the performance totally relies on the general matching ability of the siamese network. However, this manner cannot capture the temporal variations of targets or background clutter. In this work, we propose a novel gradient-guided network to exploit the discriminative information in gradients and update the template in the siamese network through feed-forward and backward operations. Our algorithm performs fee…
Template-matching methods for visual tracking have gained popularity recently due to their good performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. The reading and writing process of the external memory is controlled by an LSTM network with the search feature map as input. A spatial attention mechanism is applied to concentrate the LSTM inpu…
Nanjing University of Science and Technology（南京理工大学）
Siamese network based trackers formulate 3D single object tracking as cross-correlation learning between point features of a template and a search area. Due to the large appearance variation between the template and search area during tracking, how to learn the robust cross correlation between them for identifying the potential target in the search area is still a challenging problem. In this paper, we explicitly use Transformer to form a 3D Siamese Transformer network for learning robust cross correlation between the template and the search area of point clouds. Specifically, we develop a Sia…
Siamese-based trackers have achieved excellent performance on visual object tracking. However, the target template is not updated online, and the features of the target template and search image are computed independently in a Siamese architecture. In this paper, we propose Deformable Siamese Attention Networks, referred to as SiamAttn, by introducing a new Siamese attention mechanism that computes deformable self-attention and cross-attention. The self attention learns strong context information via spatial attention, and selectively emphasizes interdependent channel-wise features with channe…
In a point cloud sequence, 3D object tracking aims to predict the location and orientation of an object in the current search point cloud given a template point cloud. Motivated by the success of transformers, we propose Point Tracking TRansformer (PTTR), which efficiently predicts high-quality 3D tracking results in a coarse-to-fine manner with the help of transformer operations. PTTR consists of three novel designs. 1) Instead of random sampling, we design Relation-Aware Sampling to preserve relevant points to given templates during subsampling. 2) Furthermore, we propose a Point Relation Tr…
Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the loc…
MambaEVT: Event Stream based Visual Object Tracking using State Space Model
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；Beijing Institute of Technology（北京理工大学）
Template-based discriminative trackers are currently the dominant tracking methods due to their robustness and accuracy, and the Siamese-network-based methods that depend on cross-correlation operation between features extracted from template and search images show the state-of-the-art tracking performance. However, general cross-correlation operation can only obtain relationship between local patches in two feature maps. In this paper, we propose a novel tracker network based on a powerful attention mechanism called Transformer encoder-decoder architecture to gain global and rich contextual i…
Accurate tracking is still a challenging task due to appearance variations, pose and view changes, and geometric deformations of target in videos. Recent anchor-free trackers provide an efficient regression mechanism but fail to produce precise bounding box estimation. To address these issues, this paper repurposes a Transformer-alike regression branch, termed as Target Transformed Regression (TREG), for accurate anchor-free tracking. The core to our TREG is to model pair-wise relation between elements in target template and search region, and use the resulted target enhanced visual representa…
Nanjing University of Science and Technology（南京理工大学）
Learning robust feature matching between the template and search area is crucial for 3D Siamese tracking. The core of Siamese feature matching is how to assign high feature similarity on the corresponding points between the template and search area for precise object localization. In this paper, we propose a novel point cloud registration-driven Siamese tracking framework, with the intuition that spatially aligned corresponding points (via 3D registration) tend to achieve consistent feature representations. Specifically, our method consists of two modules, including a tracking-specific nonloca…
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
In video object tracking, there exist rich temporal contexts among successive frames, which have been largely overlooked in existing trackers. In this work, we bridge the individual video frames and explore the temporal contexts across them via a transformer architecture for robust object tracking. Different from classic usage of the transformer in natural language processing tasks, we separate its encoder and decoder into two parallel branches and carefully design them within the Siamese-like tracking pipelines. The transformer encoder promotes the target templates via attention-based feature…
Tianjin University（天津大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；University of Science and Technology of China（中国科学技术大学）；Automotive Data of China (Tianjin) Co., Ltd（中国汽车技术研究中心有限公司）
Recently, template-based trackers have become the leading tracking algorithms with promising performance in terms of efficiency and accuracy. However, the correlation operation between query feature and the given template only exploits accurate target localization, leading to state estimation error especially when the target suffers from severe deformable variations. To address this issue, segmentation-based trackers have been proposed that employ per-pixel matching to improve the tracking performance of deformable objects effectively. However, most of existing trackers only refer to the targe…
College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, Nanjing University of Science and Technology（南京理工大学智能感知与高维信息处理教育部重点实验室）
According to observations, different visual objects have different salient features in different scenarios. Even for the same object, its salient shape and appearance features may change greatly from time to time in a long-term tracking task. Motivated by them, we proposed an end-to-end feature fusion framework based on Siamese network, named FF-Siam, which can effectively fuse different features for adaptive visual tracking. The framework consists of four layers. A feature extraction layer is designed to extract the different features of the target region and search region. The extracted feat…
Southeast University（东南大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；JD Finance America Corporation（京东金融美国公司）；Tianjin University of Technology（天津理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Lab（鹏城实验室）
Existing visual object tracking usually learns a bounding-box based template to match the targets across frames, which cannot accurately learn a pixel-wise representation, thereby being limited in handling severe appearance variations. To address these issues, much effort has been made on segmentation-based tracking, which learns a pixel-wise object-aware template and can achieve higher accuracy than bounding-box template based tracking. However, existing segmentation-based trackers are ineffective in learning the spatio-temporal correspondence across frames due to no use of the rich temporal …
Huazhong University of Science and Technology（华中科技大学）；Microsoft Research Asia（微软亚洲研究院）
Model update lies at the heart of object tracking. Generally, model update is formulated as an online learning problem where a target model is learned over the online training set. Our key innovation is to \emph{formulate the model update problem in the meta-learning framework and learn the online learning algorithm itself using large numbers of offline videos}, i.e., \emph{learning to update}. The learned updater takes as input the online training set and outputs an updated target model. As a first attempt, we design the learned updater based on recurrent neural networks (RNNs) and demonstrat…
Dalian University of Technology（大连理工大学）；University of Amsterdam（阿姆斯特丹大学）；Peng Cheng Laboratory（鹏城实验室）；China Science IntelliCloud Technology Co., Ltd（中科智能云科技有限公司）
Long-term visual tracking has drawn increasing attention because it is much closer to practical applications than short-term tracking. Most top-ranked long-term trackers adopt the offline-trained Siamese architectures, thus, they cannot benefit from great progress of short-term trackers with online update. However, it is quite risky to straightforwardly introduce online-update-based trackers to solve the long-term problem, due to long-term uncertain and noisy observations. In this work, we propose a novel offline-trained Meta-Updater to address an important but unsolved problem: Is the tracker…
School of Mechanical and Electrical Engineering, China University of Mining and Technology（中国矿业大学机电工程学院）；Jiangsu Collaborative Innovation Center of Intelligent Mining Equipment, China University of Mining and Technology（中国矿业大学江苏省智能矿山装备协同创新中心）；School of Intelligent Manufacturing, Jiangsu Vocational Institute of Architectural Technology（江苏建筑职业技术学院智能制造学院）
Computer vision has received a significant attention in recent years, which is one of the important parts for robots to apperceive external environment. Discriminative Correlation Filter (DCF) based trackers gained more popularity due to their efficiency, however, tracking in low-illumination environments is a challenging problem, not yet successfully addressed in the literature. In this work, we tackle the problems by introducing Low-Illumination Long-term Correlation Tracker (LLCT). First, fused features only including HOG and Color Names are employed to boost the tracking efficiency. Second…
This paper proposes a novel framework to alleviate the model drift problem in visual tracking, which is based on paced updates and trajectory selection. Given a base tracker, an ensemble of trackers is generated, in which each tracker's update behavior will be paced and then traces the target object forward and backward to generate a pair of trajectories in an interval. Then, we implicitly perform self-examination based on trajectory pair of each tracker and select the most robust tracker. The proposed framework can effectively leverage temporal context of sequential frames and avoid to learn …
SUIT: Spatial-Spectral Union-Intersection Interaction Network for Hyperspectral Object Tracking
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；College of Computer Science, Zhejiang University（浙江大学计算机科学学院）
OST: Efficient One-stream Network for 3D Single Object Tracking in Point Clouds
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Economics and Management, China University of Mining and Technology（中国矿业大学经济管理学院）；School of Software, Tsinghua University（清华大学软件学院）；Dalian University of Technology（大连理工大学）
Nanyang Technological University（南洋理工大学）；DSO National Laboratories（DSO国家实验室）
Robust visual tracking for long video sequences is a research area that has many important applications. The main challenges include how the target image can be modeled and how this model can be updated. In this paper, we model the target using a covariance descriptor, as this descriptor is robust to problems such as pixel-pixel misalignment, pose and illumination changes, that commonly occur in visual tracking. We model the changes in the template using a generative process. We introduce a new dynamical model for the template update using a random walk on the Riemannian manifold where the cov…
Huazhong University of Science and Technology（华中科技大学）
Towards 3D object tracking in point clouds, a novel point-to-box network termed P2B is proposed in an end-to-end learning manner. Our main idea is to first localize potential target centers in 3D search area embedded with target information. Then point-driven 3D target proposal and verification are executed jointly. In this way, the time-consuming 3D exhaustive search can be avoided. Specifically, we first sample seeds from the point clouds in template and search area respectively. Then, we execute permutation-invariant feature augmentation to embed target clues from template into search area …
An Efficient Token Compression Framework for Visual Object Tracking
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University（广西多源信息挖掘与安全重点实验室，广西师范大学）；University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University（教育智能技术大学工程研究中心，广西师范大学）；Guangxi Key Laboratory of Machine Vision and Intelligent Control, Wuzhou University（广西机器视觉与智能控制重点实验室，梧州大学）
Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking
The Hong Kong University of Science and Technology, Guangzhou（香港科技大学（广州））；Zhejiang University（浙江大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Hangzhou Dianzi University（杭州电子科技大学）
Huazhong University of Science and Technology（华中科技大学）；La Trobe University（拉筹伯大学）
Transformer framework has been showing superior performances in visual object tracking for its great strength in information aggregation across the template and search image with the well-known attention mechanism. Most recent advances focus on exploring attention mechanism variants for better information aggregation. We find these schemes are equivalent to or even just a subset of the basic self-attention mechanism. In this paper, we prove that the vanilla self-attention structure is sufficient for information aggregation, and structural adaption is unnecessary. The key is not the attention s…
Tracking is one of the most important but still difficult tasks in computer vision and pattern recognition. The main difficulties in the tracking field are appearance variation and occlusion. Most traditional tracking methods set the parameters or templates to track target objects in advance and should be modified accordingly. Thus, we propose a new and robust tracking method using a Fully Convolutional Network (FCN) to obtain an object probability map and Dynamic Programming (DP) to seek the globally optimal path through all frames of video. Our proposed method solves the object appearance va…
Zhejiang University of Technology（浙江工业大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Adelaide（阿德莱德大学）
Siamese network based trackers formulate the visual tracking task as a similarity matching problem. Almost all popular Siamese trackers realize the similarity learning via convolutional feature cross-correlation between a target branch and a search branch. However, since the size of target feature region needs to be pre-fixed, these cross-correlation base methods suffer from either reserving much adverse background information or missing a great deal of foreground information. Moreover, the global matching between the target and search region also largely neglects the target structure and part…
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Science and Technologies Facilities Council（科学与技术设施委员会）；Beijing Jiaotong University（北京交通大学）
Correlation filter has been proven to be an effective tool for a number of approaches in visual tracking, particularly for seeking a good balance between tracking accuracy and speed. However, correlation filter based models are susceptible to wrong updates stemming from inaccurate tracking results. To date, little effort has been devoted towards handling the correlation filter update problem. In this paper, we propose a novel approach to address the correlation filter update problem. In our approach, we update and maintain multiple correlation filter models in parallel, and we use deep reinfor…
In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking
School of Cyber Science and Engineering, Qufu Normal University（曲阜师范大学网络与网络安全学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；School of Integrated Circuits, Harbin Institute of Technology（哈尔滨工业大学集成电路学院）；Faculty of Software and Information Science, Iwate Prefectural University（岩手县立大学软件与信息科学学部）
NICTA（国家信息与通信技术研究院）；Australian National University（澳大利亚国立大学）；University of Adelaide（阿德莱德大学）
In this work we generalize the plain MS trackers and attempt to overcome standard mean shift trackers' two limitations. It is well known that modeling and maintaining a representation of a target object is an important component of a successful visual tracker. However, little work has been done on building a robust template model for kernel-based MS tracking. In contrast to building a template from a single frame, we train a robust object representation model from a large amount of data. Tracking is viewed as a binary classification problem, and a discriminative classification rule is learned …
Munich School of Robotics and Machine Intelligence（慕尼黑机器人与机器智能学院）；Technical University of Munich（慕尼黑工业大学）
Recent advances in visual tracking are based on siamese feature extractors and template matching. For this category of trackers, latest research focuses on better feature embeddings and similarity measures. In this work, we focus on building holistic object representations for tracking. We propose a framework that is designed to be used on top of previous trackers without any need for further training of the siamese network. The framework leverages the idea of obtaining additional object templates during the tracking process. Since the number of stored templates is limited, our method only kee…
The problem of robust extraction of visual odometry from a sequence of images obtained by an eye in hand camera configuration is addressed. A novel approach toward solving planar template based tracking is proposed which performs a non-linear image alignment for successful retrieval of camera transformations. In order to obtain global optimum a bio-metaheuristic is used for optimization of similarity among the planar regions. The proposed method is validated on image sequences with real as well as synthetic transformations and found to be resilient to intensity variations. A comparative analys…
Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-spec…
SBMVTrack: Spike-Budgeted Multi-View Learning for Power-Efficient UAV Tracking
College of Computer Science and Engineering, Guilin University of Technology（桂林理工大学计算机科学与工程学院）；School of Information Engineering, Wuhan University of Technology（武汉理工大学信息工程学院）
RWTH Aachen University（亚琛工业大学）；University of Oxford（牛津大学）
We present Siam R-CNN, a Siamese re-detection architecture which unleashes the full power of two-stage object detection approaches for visual object tracking. We combine this with a novel tracklet-based dynamic programming algorithm, which takes advantage of re-detections of both the first-frame template and previous-frame predictions, to model the full history of both the object to be tracked and potential distractor objects. This enables our approach to make better tracking decisions, as well as to re-detect tracked objects after long occlusion. Finally, we propose a novel hard example minin…
SwiTrack: Tri-State Switch for Cross-Modal Object Tracking
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；School of Mathematics, Southeast University（东南大学数学学院）；Purple Mountain Laboratories（紫金山实验室）
