Multi-object tracking (MOT) is a challenging practical problem for vision based applications. Most recent approaches for MOT use precomputed detections from models such as Faster RCNN, performing fine-tuning of bounding boxes and association in subsequent phases. However, this is not suitable for actual industrial applications due to unavailability of detections upfront. In their recent work, Wang et al. proposed a tracking pipeline that uses a Joint detection and embedding model and performs target localization and association in realtime. Upon investigating the tracking by detection paradigm, we find that the tracking pipeline can be made faster by performing localization and association tasks parallely with model prediction. This, and other computational optimizations such as using mixed precision model and performing batchwise detection result in a speed-up of the tracking pipeline by 57.8\% (19 FPS to 30 FPS) on FullHD resolution. Moreover, the speed is independent of the object density in image sequence. The main contribution of this paper is showcasing a generic pipeline which can be used to speed up detection based object tracking methods. We also reviewed different batch sizes for optimal performance, taking into consideration GPU memory usage and speed.
Tsinghua University（清华大学）；Graduate School at Shenzhen, Tsinghua University（清华大学深圳研究生院）
Discriminatively learned correlation filters (DCF) have been widely used in online visual tracking filed due to its simplicity and efficiency. These methods utilize a periodic assumption of the training samples to construct a circulant data matrix, which implicitly increases the training samples and reduces both storage and computational complexity.The periodic assumption also introduces unwanted boundary effects. Recently, Spatially Regularized Correlation Filters (SRDCF) solved this issue by introducing penalization on correlation filter coefficients depending on their spatial location. Howe…
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）
Single object tracking (SOT) is currently one of the most important tasks in computer vision. With the development of the deep network and the release for a series of large scale datasets for single object tracking, siamese networks have been proposed and perform better than most of the traditional methods. However, recent siamese networks get deeper and slower to obtain better performance. Most of these methods could only meet the needs of real-time object tracking in ideal environments. In order to achieve a better balance between efficiency and accuracy, we propose a simpler siamese network…
AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；Computer Vision Center（计算机视觉中心）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
In video object tracking, there exist rich temporal contexts among successive frames, which have been largely overlooked in existing trackers. In this work, we bridge the individual video frames and explore the temporal contexts across them via a transformer architecture for robust object tracking. Different from classic usage of the transformer in natural language processing tasks, we separate its encoder and decoder into two parallel branches and carefully design them within the Siamese-like tracking pipelines. The transformer encoder promotes the target templates via attention-based feature…
Microsoft Research Asia（微软亚洲研究院）；Dalian University of Technology（大连理工大学）；Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）
Object tracking has achieved significant progress over the past few years. However, state-of-the-art trackers become increasingly heavy and expensive, which limits their deployments in resource-constrained applications. In this work, we present LightTrack, which uses neural architecture search (NAS) to design more lightweight and efficient object trackers. Comprehensive experiments show that our LightTrack is effective. It can find trackers that achieve superior performance compared to handcrafted SOTA trackers, such as SiamRPN++ and Ocean, while using much fewer model Flops and parameters. Mo…
The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）；ByteDance Inc.（字节跳动公司）
A typical pipeline for multi-object tracking (MOT) is to use a detector for object localization, and following re-identification (re-ID) for object association. This pipeline is partially motivated by recent progress in both object detection and re-ID, and partially motivated by biases in existing tracking datasets, where most objects tend to have distinguishing appearance and re-ID models are sufficient for establishing associations. In response to such bias, we would like to re-emphasize that methods for multi-object tracking should also work when object appearance is not sufficiently discri…
Queensland University of Technology（昆士兰科技大学）；University of Sydney（悉尼大学）
This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our trac…
Politecnico di Milano（米兰理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Guilan（吉兰大学）
Recently, sparse representation based visual tracking methods have attracted increasing attention in the computer vision community. Although achieve superior performance to traditional tracking methods, however, a basic problem has not been answered yet --- that whether the sparsity constrain is really needed for visual tracking? To answer this question, in this paper, we first propose a robust non-sparse representation based tracker and then conduct extensive experiments to compare it against several state-of-the-art sparse representation based trackers. Our experiment results and analysis in…
The design of more complex and powerful neural network models has significantly advanced the state-of-the-art in visual object tracking. These advances can be attributed to deeper networks, or the introduction of new building blocks, such as transformers. However, in the pursuit of increased tracking performance, runtime is often hindered. Furthermore, efficient tracking architectures have received surprisingly little attention. In this paper, we introduce the Exemplar Transformer, a transformer module utilizing a single instance level attention layer for realtime visual object tracking. E.T.T…
Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-spec…
Detection and tracking of dynamic objects is a key feature for autonomous behavior in a continuously changing environment. With the increasing popularity and capability of micro aerial vehicles (MAVs) efficient algorithms have to be utilized to enable multi object tracking on limited hardware and data provided by lightweight sensors. We present a novel segmentation approach based on a combination of median filters and an efficient pipeline for detection and tracking of small objects within sparse point clouds generated by a Velodyne VLP-16 sensor. We achieve real-time performance on a single c…
With the recent advances in the object detection research field, tracking-by-detection has become the leading paradigm adopted by multi-object tracking algorithms. By extracting different features from detected objects, those algorithms can estimate the objects' similarities and association patterns along successive frames. However, since similarity functions applied by tracking algorithms are handcrafted, it is difficult to employ them in new contexts. In this study, it is investigated the use of artificial neural networks to learning a similarity function that can be used among detections. D…
3D multi-object tracking in LiDAR point clouds is a key ingredient for self-driving vehicles. Existing methods are predominantly based on the tracking-by-detection pipeline and inevitably require a heuristic matching step for the detection association. In this paper, we present SimTrack to simplify the hand-crafted tracking paradigm by proposing an end-to-end trainable model for joint detection and tracking from raw point clouds. Our key design is to predict the first-appear location of each object in a given snippet to get the tracking identity and then update the location based on motion est…
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Multi-Object Tracking over humans has improved rapidly with the development of object detection and re-identification. However, multi-actor tracking over humans with similar appearance and nonlinear movement can still be very challenging even for the state-of-the-art tracking algorithm. Current motion-based tracking algorithms often use Kalman Filter to predict the motion of an object, however, its linear movement assumption can cause failure in tracking when the target is not moving linearly. And for multi-players tracking over the sports field, because the players in the same team are usuall…
Traditional multiple object tracking methods divide the task into two parts: affinity learning and data association. The separation of the task requires to define a hand-crafted training goal in affinity learning stage and a hand-crafted cost function of data association stage, which prevents the tracking goals from learning directly from the feature. In this paper, we present a new multiple object tracking (MOT) framework with data-driven association method, named as Tracklet Association Tracker (TAT). The framework aims at gluing feature learning and data association into a unity by a bi-lev…
In this paper, we propose a pipeline for multi-target visual tracking under multi-camera system. For multi-camera system tracking problem, efficient data association across cameras, and at the same time, across frames becomes more important than single-camera system tracking. However, most of the multi-camera tracking algorithms emphasis on single camera across frame data association. Thus in our work, we model our tracking problem as a global graph, and adopt Generalized Maximum Multi Clique optimization problem as our core algorithm to take both across frame and across camera data correlatio…
Dalian University of Technology（大连理工大学）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we present a new tracking architecture with an encoder-decoder transformer as the key component. The encoder models the global spatio-temporal feature dependencies between target objects and search regions, while the decoder learns a query embedding to predict the spatial positions of the target objects. Our method casts object tracking as a direct bounding box prediction problem, without using any proposals or predefined anchors. With the encoder-decoder transformer, the prediction of objects just uses a simple fully-convolutional network, which estimates the corners of objects…
In dynamic environments, performance of visual SLAM techniques can be impaired by visual features taken from moving objects. One solution is to identify those objects so that their visual features can be removed for localization and mapping. This paper presents a simple and fast pipeline that uses deep neural networks, extended Kalman filters and visual SLAM to improve both localization and mapping in dynamic environments (around 14 fps on a GTX 1080). Results on the dynamic sequences from the TUM dataset using RTAB-Map as visual SLAM suggest that the approach achieves similar localization per…
The University of Georgia（佐治亚大学）；University of Vermont（佛蒙特大学）
Tracking cell particles in 3D microscopy videos is a challenging task but is of great significance for modeling the motion of cells. Proper characterization of the cell's shape, evolution, and their movement over time is crucial to understanding and modeling the mechanobiology of cell migration in many diseases. One in particular, toxoplasmosis is the disease caused by the parasite Toxoplasma gondii. Roughly, one-third of the world's population tests positive for T. gondii. Its virulence is linked to its lytic cycle, predicated on its motility and ability to enter and exit nucleated cells; the…
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
Recent online Multi-Object Tracking (MOT) methods have achieved desirable tracking performance. However, the tracking speed of most existing methods is rather slow. Inspired from the fact that the adjacent frames are highly relevant and redundant, we divide the frames into key and non-key frames respectively and track objects in the compressed domain. For the key frames, the RGB images are restored for detection and data association. To make data association more reliable, an appearance Convolutional Neural Network (CNN) which can be jointly trained with the detector is proposed. For the non-k…
In Computer Vision domain, moving Object Tracking considered as one of the toughest problem.As there so many factors associated like illumination of light, noise, occlusion, sudden start and stop of moving object, shading which makes tracking even harder problem not only for dynamic background but also for static background.In this paper we present a new object tracking algorithm based on Dominant points on tracked object using Quantum particle swarm optimization (QPSO) which is a new different version of PSO based on Quantum theory. The novelty in our approach is that it can be successfully a…
Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for t…
Hong Kong University of Science and Technology（香港科技大学）
A robust 3D object tracker which continuously tracks surrounding objects and estimates their trajectories is key for self-driving vehicles. Most existing tracking methods employ a tracking-by-detection strategy, which usually requires complex pair-wise similarity computation and neglects the nature of continuous object motion. In this paper, we propose to directly learn 3D object correspondences from temporal point cloud data and infer the motion information from correspondence patterns. We modify the standard 3D object detector to process two lidar frames at the same time and predict bounding…
University of California at Santa Barbara（加州大学圣塔芭芭拉分校）；Samsung Research America（三星美国研究院）
In this paper we introduce a fully end-to-end approach for visual tracking in videos that learns to predict the bounding box locations of a target object at every frame. An important insight is that the tracking problem can be considered as a sequential decision-making process and historical semantics encode highly relevant information for future decisions. Based on this intuition, we formulate our model as a recurrent convolutional neural network agent that interacts with a video overtime, and our model can be trained with reinforcement learning (RL) algorithms to learn good tracking policies…
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
3D Multi-object tracking (MOT) is crucial to autonomous systems. Recent work often uses a tracking-by-detection pipeline, where the feature of each object is extracted independently to compute an affinity matrix. Then, the affinity matrix is passed to the Hungarian algorithm for data association. A key process of this pipeline is to learn discriminative features for different objects in order to reduce confusion during data association. To that end, we propose two innovative techniques: (1) instead of obtaining the features for each object independently, we propose a novel feature interaction …
Università degli Studi di Genova（热那亚大学）；MaLGa Machine Learning Genoa Center（MaLGa热那亚机器学习中心）
Object detection and tracking in videos represent essential and computationally demanding building blocks for current and future visual perception systems. In order to reduce the efficiency gap between available methods and computational requirements of real-world applications, we propose to re-think one of the most successful methods for image object detection, Faster R-CNN, and extend it to the video domain. Specifically, we extend the detection framework to learn instance-level embeddings which prove beneficial for data association and re-identification purposes. Focusing on the computation…
Shanghai Grandhonor Information Technology Co.Ltd（上海宏景信息科技有限公司）；Nanjing University of Aeronautics and Astronautic（南京航空航天大学）；New York University（纽约大学）
Multiple Object Tracking (MOT) has been a useful yet challenging task in many real-world applications such as video surveillance, intelligent retail, and smart city. The challenge is how to model long-term temporal dependencies in an efficient manner. Some recent works employ Recurrent Neural Networks (RNN) to obtain good performance, which, however, requires a large amount of training data. In this paper, we proposed a novel tracking method that integrates the auto-tuning Kalman method for prediction and the Gated Recurrent Unit (GRU) and achieves a near-optimum with a small amount of trainin…
We present a real-time on-device hand tracking pipeline that predicts hand skeleton from single RGB camera for AR/VR applications. The pipeline consists of two models: 1) a palm detector, 2) a hand landmark model. It's implemented via MediaPipe, a framework for building cross-platform ML solutions. The proposed model and pipeline architecture demonstrates real-time inference speed on mobile GPUs and high prediction quality. MediaPipe Hands is open sourced at https://mediapipe.dev.
Prior work presented the sentence tracker, a method for scoring how well a sentence describes a video clip or alternatively how well a video clip depicts a sentence. We present an improved method for optimizing the same cost function employed by this prior work, reducing the space complexity from exponential in the sentence length to polynomial, as well as producing a qualitatively identical result in time polynomial in the sentence length instead of exponential. Since this new method is plug-compatible with the prior method, it can be used for the same applications: video retrieval with sente…
Recently, single-stage embedding based deep learning algorithms gain increasing attention in cell segmentation and tracking. Compared with the traditional "segment-then-associate" two-stage approach, a single-stage algorithm not only simultaneously achieves consistent instance cell segmentation and tracking but also gains superior performance when distinguishing ambiguous pixels on boundaries and overlaps. However, the deployment of an embedding based algorithm is restricted by slow inference speed (e.g., around 1-2 mins per frame). In this study, we propose a novel Faster Mean-shift algorithm…
Peking University（北京大学）；Beijing Institute for General AI（北京通用人工智能研究院）；Virginia Tech（弗吉尼亚理工大学）；Stanford University（斯坦福大学）；Tsinghua University（清华大学）；Columbia University（哥伦比亚大学）
In this work, we tackle the challenging task of jointly tracking hand object pose and reconstructing their shapes from depth point cloud sequences in the wild, given the initial poses at frame 0. We for the first time propose a point cloud based hand joint tracking network, HandTrackNet, to estimate the inter-frame hand joint motion. Our HandTrackNet proposes a novel hand pose canonicalization module to ease the tracking task, yielding accurate and robust hand joint tracking. Our pipeline then reconstructs the full hand via converting the predicted hand joints into a template-based parametric …
We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Track…
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Sydney Nano Institute（悉尼纳米研究所）
Cell segmentation and tracking in microscopy images are of great significance to new discoveries in biology and medicine. In this study, we propose a novel approach to combine cell segmentation and cell tracking into a unified end-to-end deep learning based framework, where cell detection and segmentation are performed with a current instance segmentation pipeline and cell tracking is implemented by integrating Siamese Network with the pipeline. Besides, tracking performance is improved by incorporating spatial information into the network and fusing spatial and visual prediction. Our approach…
Tracking a crowd in 3D using multiple RGB cameras is a challenging task. Most previous multi-camera tracking algorithms are designed for offline setting and have high computational complexity. Robust real-time multi-camera 3D tracking is still an unsolved problem. In this work, we propose a novel end-to-end tracking pipeline, Deep Multi-Camera Tracking (DMCT), which achieves reliable real-time multi-camera people tracking. Our DMCT consists of 1) a fast and novel perspective-aware Deep GroudPoint Network, 2) a fusion procedure for ground-plane occupancy heatmap estimation, 3) a novel Deep Glim…
Deep SORT\cite{wojke2017simple} is a tracking-by-detetion approach to multiple object tracking with a detector and a RE-ID model. Both separately training and inference with the two model is time-comsuming. In this paper, we unify the detector and RE-ID model into an end-to-end network, by adding an additional track branch for tracking in Faster RCNN architecture. With a unified network, we are able to train the whole model end-to-end with multi loss, which has shown much benefit in other recent works. The RE-ID model in Deep SORT needs to use deep CNNs to extract feature map from detected obj…
In the past decade many robots were deployed in the wild, and people detection and tracking is an important component of such deployments. On top of that, one often needs to run modules which analyze persons and extract higher level attributes such as age and gender, or dynamic information like gaze and pose. The latter ones are especially necessary for building a reactive, social robot-person interaction. In this paper, we combine those components in a fully modular detection-tracking-analysis pipeline, called DetTA. We investigate the benefits of such an integration on the example of head an…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；MPI Tübingen（图宾根马克斯·普朗克研究所）；University of Toronto（多伦多大学）
One of the most popular approaches to multi-target tracking is tracking-by-detection. Current min-cost flow algorithms which solve the data association problem optimally have three main drawbacks: they are computationally expensive, they assume that the whole video is given as a batch, and they scale badly in memory and computation with the length of the video sequence. In this paper, we address each of these issues, resulting in a computationally and memory-bounded solution. First, we introduce a dynamic version of the successive shortest-path algorithm which solves the data association probl…
Fraunhofer HHI（弗劳恩霍夫电信研究所）；Humboldt University of Berlin（柏林洪堡大学）
In this paper, we present a multi-object 6D detection and tracking pipeline for potentially similar and non-textured objects. The combination of a convolutional neural network for object classification and rough pose estimation with a local pose refinement and an automatic mismatch detection enables direct application in real-time AR scenarios. A new network architecture, trained solely with synthetic images, allows simultaneous pose estimation of multiple objects with reduced GPU memory consumption and enhanced performance. In addition, the pose estimates are further improved by a local edge-…
Peking University（北京大学）；Stanford University（斯坦福大学）；Shandong University（山东大学）；UCSD（加州大学圣地亚哥分校）；Tencent AI Lab（腾讯AI实验室）；Beijing Institute for General AI（北京通用人工智能研究院）
In this work, we tackle the problem of category-level online pose tracking of objects from point cloud sequences. For the first time, we propose a unified framework that can handle 9DoF pose tracking for novel rigid object instances as well as per-part pose tracking for articulated objects from known categories. Here the 9DoF pose, comprising 6D pose and 3D size, is equivalent to a 3D amodal bounding box representation with free 6D pose. Given the depth point cloud at the current frame and the estimated pose from the last frame, our novel end-to-end pipeline learns to accurately update the pos…
Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning
University of Southern California（南加州大学）；Meituan（美团）；Sun Yat-sen University（中山大学）；University of Shanghai for Science and Technology（上海理工大学）；Southeast University（东南大学）
Safran Electronics and Defense（赛峰电子与防务公司）；Normandie Univ, UNICAEN, ENSICAEN, CNRS（诺曼底大学、卡昂大学、卡昂国立高等工程师学校、法国国家科学研究中心）
Detecting small vehicles in aerial images is a difficult job that can be challenging even for humans. Rotating objects, low resolution, small inter-class variability and very large images comprising complicated backgrounds render the work of photo-interpreters tedious and wearisome. Unfortunately even the best classical detection pipelines like Faster R-CNN cannot be used off-the-shelf with good results because they were built to process object centric images from day-to-day life with multi-scale vertical objects. In this work we build on the Faster R-CNN approach to turn it into a detection f…
Kyungpook National University（庆北国立大学）；Indian Statistical Institute（印度统计研究所）
In machine learning and computer vision, mean shift (MS) qualifies as one of the most popular mode-seeking algorithms used for clustering and image segmentation. It iteratively moves each data point to the weighted mean of its neighborhood data points. The computational cost required to find the neighbors of each data point is quadratic to the number of data points. Consequently, the vanilla MS appears to be very slow for large-scale datasets. To address this issue, we propose a mode-seeking algorithm called GridShift, with significant speedup and principally based on MS. To accelerate, GridSh…
Georgia Institute of Technology（佐治亚理工学院）；Aison Pte. Ltd.（Aison私人有限公司）
Lane detection is an integral part of control systems in autonomous vehicles and lane departure warning systems as lanes are a key component of the operating environment for road vehicles. In a previous paper, a robust neural network output enhancement for active lane detection (RONELD) method augmenting deep learning lane detection models to improve active, or ego, lane accuracy performance was presented. This paper extends the work by further investigating the lane tracking methods used to increase robustness of the method to lane changes and different lane dimensions (e.g. lane marking thic…
Toyota Research Institute（丰田研究院）；Technical University of Munich（慕尼黑工业大学）；Siemens Corporate Research（西门子公司研究部）
We present a novel method to track 3D models in color and depth data. To this end, we introduce approximations that accelerate the state-of-the-art in region-based tracking by an order of magnitude while retaining similar accuracy. Furthermore, we show how the method can be made more robust in the presence of depth data and consequently formulate a new joint contour and ICP tracking energy. We present better results than the state-of-the-art while being much faster then most other methods and achieving all of the above on a single CPU core.
