We present a challenging and realistic novel dataset for evaluating 6-DOF object tracking algorithms. Existing datasets show serious limitations---notably, unrealistic synthetic data, or real data with large fiducial markers---preventing the community from obtaining an accurate picture of the state-of-the-art. Using a data acquisition pipeline based on a commercial motion capture system for acquiring accurate ground truth poses of real objects with respect to a Kinect V2 camera, we build a dataset which contains a total of 297 calibrated sequences. They are acquired in three different scenarios to evaluate the performance of trackers: stability, robustness to occlusion and accuracy during challenging interactions between a person and the object. We conduct an extensive study of a deep 6-DOF tracking architecture and determine a set of optimal parameters. We enhance the architecture and the training methodology to train a 6-DOF tracker that can robustly generalize to objects never seen during training, and demonstrate favorable performance compared to previous approaches trained specifically on the objects to track.
Visual tracking algorithms are naturally adopted in various applications, there have been several benchmarks and many tracking algorithms, more expected to appear in the future. In this report, I focus on single object tracking and revisit the details of tracker evaluation based on widely used OTB\cite{otb} benchmark by introducing a simpler, accurate, and extensible method for tracker evaluation and comparison. Experimental results suggest that there may not be an absolute winner among tracking algorithms. We have to perform detailed analysis to select suitable trackers for use cases.
University of Ljubljana（卢布尔雅那大学）；Czech Technical University in Prague（布拉格捷克理工大学）
We propose a new long-term tracking performance evaluation methodology and present a new challenging dataset of carefully selected sequences with many target disappearances. We perform an extensive evaluation of six long-term and nine short-term state-of-the-art trackers, using new performance measures, suitable for evaluating long-term tracking - tracking precision, recall and F-score. The evaluation shows that a good model update strategy and the capability of image-wide re-detection are critical for long-term tracking performance. We integrated the methodology in the VOT toolkit to automate…
6-DoF Object Tracking with Event-based Optical Flow and Frames
Humanoid Sensing and Perception, Istituto Italiano di Tecnologia（人形感知与感知，意大利技术研究所）；Event-driven Perception for Robotics, Istituto Italiano di Tecnologia（机器人驱动感知，意大利技术研究所）；University of Genoa（热那亚大学）
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Singapore Management University（新加坡管理大学）
Most of existing correlation filter-based tracking approaches only estimate simple axis-aligned bounding boxes, and very few of them is capable of recovering the underlying similarity transformation. To tackle this challenging problem, in this paper, we propose a new correlation filter-based tracker with a novel robust estimation of similarity transformation on the large displacements. In order to efficiently search in such a large 4-DoF space in real-time, we formulate the problem into two 2-DoF sub-problems and apply an efficient Block Coordinates Descent solver to optimize the estimation re…
This paper presents two visual trackers from the different paradigms of learning and registration based tracking and evaluates their application in image based visual servoing. They can track object motion with four degrees of freedom (DoF) which, as we will show here, is sufficient for many fine manipulation tasks. One of these trackers is a newly developed learning based tracker that relies on learning discriminative correlation filters while the other is a refinement of a recent 8 DoF RANSAC based tracker adapted with a new appearance model for tracking 4 DoF motion. Both trackers are shown…
Université Laval（拉瓦尔大学）；Thales Digital Solutions（泰雷兹数字解决方案）；Inria（法国国家信息与自动化研究所）
Augmented reality devices require multiple sensors to perform various tasks such as localization and tracking. Currently, popular cameras are mostly frame-based (e.g. RGB and Depth) which impose a high data bandwidth and power usage. With the necessity for low power and more responsive augmented reality systems, using solely frame-based sensors imposes limits to the various algorithms that needs high frequency data from the environement. As such, event-based sensors have become increasingly popular due to their low power, bandwidth and latency, as well as their very high frequency data acquisi…
Tracking object poses in 3D is a crucial building block for Augmented Reality applications. We propose an instant motion tracking system that tracks an object's pose in space (represented by its 3D bounding box) in real-time on mobile devices. Our system does not require any prior sensory calibration or initialization to function. We employ a deep neural network to detect objects and estimate their initial 3D pose. Then the estimated pose is tracked using a robust planar tracker. Our tracker is capable of performing relative-scale 9-DoF tracking in real-time on mobile devices. By combining use…
Colledge of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Key Research Center for Surface Monitoring and Analysis of Cultural Relics (SMARC), State Administration of Cultural Heritage, China（国家文物局文物表面监测与分析重点研究中心）；Department of Computer Science and Engineering, University of South Carolina（南卡罗来纳大学计算机科学与工程系）
Intuitively, motion blur may hurt the performance of visual object tracking. However, we lack quantitative evaluation of tracker robustness to different levels of motion blur. Meanwhile, while image deblurring methods can produce visually clearer videos for pleasing human eyes, it is unknown whether visual object tracking can benefit from image deblurring or not. In this paper, we address these two problems by constructing a Blurred Video Tracking benchmark, which contains a variety of videos with different levels of motion blurs, as well as ground truth tracking results for evaluating tracker…
Kyungpook National University（庆北国立大学）；Qatar University（卡塔尔大学）
Visual object tracking is an important computer vision problem with numerous real-world applications including human-computer interaction, autonomous vehicles, robotics, motion-based recognition, video indexing, surveillance and security. In this paper, we aim to extensively review the latest trends and advances in the tracking algorithms and evaluate the robustness of trackers in the presence of noise. The first part of this work comprises a comprehensive survey of recently proposed tracking algorithms. We broadly categorize trackers into correlation filter based trackers and the others as no…
Evaluating tracking model performance is a complicated task, particularly for non-contiguous, multi-object trackers that are crucial in defense applications. While there are various excellent tracking benchmarks available, this work expands them to quantify the performance of long-term, non-contiguous, multi-object and detection model assisted trackers. We propose a suite of MONCE (Multi-Object Non-Contiguous Entities) image tracking metrics that provide both objective tracking model performance benchmarks as well as diagnostic insight for driving tracking model development in the form of Expe…
RWTH Aachen University（亚琛工业大学）；Technical University Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（图宾根大学）
Multi-Object Tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association. To address this, we present a novel MOT evaluation metric, HOTA (Higher Order Tracking Accuracy), which explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately, which enables clear analysis of tracking performance. We evaluate the effectiven…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Despite the numerous developments in object tracking, further development of current tracking algorithms is limited by small and mostly saturated datasets. As a matter of fact, data-hungry trackers based on deep-learning currently rely on object detection datasets due to the scarcity of dedicated large-scale tracking datasets. In this work, we present TrackingNet, the first large-scale dataset and benchmark for object tracking in the wild. We provide more than 30K videos with more than 14 million dense bounding box annotations. Our dataset covers a wide selection of object classes in broad and…
State-of-the-art Multiple Object Tracking (MOT) approaches have shown remarkable performance when trained and evaluated on current benchmarks. However, these benchmarks primarily consist of clear weather scenarios, overlooking adverse atmospheric conditions such as fog, haze, smoke and dust. As a result, the robustness of trackers against these challenging conditions remains underexplored. To address this gap, we introduce physics-based volumetric fog simulation method for arbitrary MOT datasets, utilizing frame-by-frame monocular depth estimation and a fog formation optical model. We enhance …
Towards Real-World Aerial Vision Guidance with Categorical 6D Pose Tracker
Hunan University（湖南大学）；National Engineering Research Centre for Robot Visual Perception and Control（国家机器人视觉感知与控制技术工程研究中心）；Nanyang Technological University（南洋理工大学）
Hong Kong University of Science and Technology（香港科技大学）；Tencent Inc（腾讯公司）
A monocular 3D object tracking system generally has only up-to-scale pose estimation results without any prior knowledge of the tracked object. In this paper, we propose a novel idea to recover the metric scale of an arbitrary dynamic object by optimizing the trajectory of the objects in the world frame, without motion assumptions. By introducing an additional constraint in the time domain, our monocular visual-inertial tracking system can obtain continuous six degree of freedom (6-DoF) pose estimation without scale ambiguity. Our method requires neither fixed multi-camera nor depth sensor set…
Full body trackers are utilized for surveillance and security purposes, such as person-tracking robots. In the Middle East, uniform crowd environments are the norm which challenges state-of-the-art trackers. Despite tremendous improvements in tracker technology documented in the past literature, these trackers have not been trained using a dataset that captures these environments. In this work, we develop an annotated dataset with one specific target per video in a uniform crowd environment. The dataset was generated in four different scenarios where mainly the target was moving alongside the …
Cortexica Vision System Ltd（Cortexica视觉系统有限公司）；University College London（伦敦大学学院）；The University of Adelaide（阿德莱德大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）
In this paper, we introduce a variation of a state-of-the-art real-time tracker (CFNet), which adds to the original algorithm robustness to target loss without a significant computational overhead. The new method is based on the assumption that the feature map can be used to estimate the tracking confidence more accurately. When the confidence is low, we avoid updating the object's position through the feature map; instead, the tracker passes to a single-frame failure mode, during which the patch's low-level visual content is used to swiftly update the object's position, before recovering from…
Robust Single Object Tracking in LiDAR Point Clouds under Adverse Weather Conditions
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Economics and Management, China University of Mining and Technology（中国矿业大学经济管理学院）；DUT-BSU Joint Institute, Dalian University of Technology（大连理工大学-白俄罗斯国立大学联合学院）
Evaluating Sensitivity Parameters in Smartphone-Based Gaze Estimation: A Comparative Study of Appearance-Based and Infrared Eye Trackers
School of Computer, Data and Mathematical Sciences, Western Sydney University, Penrith, New South Wales, Australia（计算机、数据与数学科学学院，西悉尼大学，彭里，新南威尔士州，澳大利亚）；The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, Australia（脑行为与发育研究所，西悉尼大学，彭里，新南威尔士州，澳大利亚）；School of Systems and Computing, University of New South Wales, Canberra, ACT, Australia（系统与计算学院，新南威尔士大学，堪培拉，澳大利亚）
MVTec Software GmbH（MVTec软件有限公司）；Technical University of Munich (TUM)（慕尼黑工业大学（TUM））
The accuracy of object detectors and trackers is most commonly evaluated by the Intersection over Union (IoU) criterion. To date, most approaches are restricted to axis-aligned or oriented boxes and, as a consequence, many datasets are only labeled with boxes. Nevertheless, axis-aligned or oriented boxes cannot accurately capture an object's shape. To address this, a number of densely segmented datasets has started to emerge in both the object detection and the object tracking communities. However, evaluating the accuracy of object detectors and trackers that are restricted to boxes on densely…
We present a real-time approach for image-based localization within large scenes that have been reconstructed offline using structure from motion (Sfm). From monocular video, our method continuously computes a precise 6-DOF camera pose, by efficiently tracking natural features and matching them to 3D points in the Sfm point cloud. Our main contribution lies in efficiently interleaving a fast keypoint tracker that uses inexpensive binary feature descriptors with a new approach for direct 2D-to-3D matching. The 2D-to-3D matching avoids the need for online extraction of scale-invariant features. …
DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
DRGBT-1K: A Large-scale High-quality Benchmark for Dynamic RGBT Tracking
Anhui University（安徽大学）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电子信息采集与保护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
We propose a novel part-based method for tracking an arbitrary object in challenging video sequences. The colour distribution of tracked image patches on the target object are represented by pairs of RGB samples and counts of how many pixels in the patch are similar to them. Patches are placed by segmenting the object in the given bounding box and placing patches in homogeneous regions of the object. These are located in subsequent image frames by applying non-shearing affine transformations to the patches' previous locations, locally optimising the best of these, and evaluating their quality …
FEMOT: Multi-Object Tracking using Frame and Event Cameras
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理全国重点实验室）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
South China University of Technology（华南理工大学）；Samsung, USA（三星美国）
Learning robotic grasps from visual observations is a promising yet challenging task. Recent research shows its great potential by preparing and learning from large-scale synthetic datasets. For the popular, 6 degree-of-freedom (6-DOF) grasp setting of parallel-jaw gripper, most of existing methods take the strategy of heuristically sampling grasp candidates and then evaluating them using learned scoring functions. This strategy is limited in terms of the conflict between sampling efficiency and coverage of optimal grasps. To this end, we propose in this work a novel, end-to-end \emph{Grasp Pr…
Adaptive tracking-by-detection approaches are popular for tracking arbitrary objects. They treat the tracking problem as a classification task and use online learning techniques to update the object model. However, these approaches are heavily invested in the efficiency and effectiveness of their detectors. Evaluating a massive number of samples for each frame (e.g., obtained by a sliding window) forces the detector to trade the accuracy in favor of speed. Furthermore, misclassification of borderline samples in the detector introduce accumulating errors in tracking. In this study, we propose a…
COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作 medianet 创新中心）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；CloudWalk Technology Co., Ltd（云从科技股份有限公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；Computer Vision Laboratory, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；Inception Institute of Artificial Intelligence（inception人工智能研究院）
We propose an end-to-end tracking framework for fusing the RGB and TIR modalities in RGB-T tracking. Our baseline tracker is DiMP (Discriminative Model Prediction), which employs a carefully designed target prediction network trained end-to-end using a discriminative loss. We analyze the effectiveness of modality fusion in each of the main components in DiMP, i.e. feature extractor, target estimation network, and classifier. We consider several fusion mechanisms acting at different levels of the framework, including pixel-level, feature-level and response-level. Our tracker is trained in an en…
MultiDiffSense: Diffusion-Based Multi-Modal Visuo-Tactile Image Generation Conditioned on Object Shape and Contact Pose
Department of Bioengineering, Imperial-X Initiative, Imperial College London（生物工程系、Imperial-X计划、帝国理工学院伦敦分校）；CAMS-Oxford Institute, University of Oxford（CAMS-牛津研究所、牛津大学）
Intelligent manipulation benefits from the capacity to flexibly control an end-effector with high degrees of freedom (DoF) and dynamically react to the environment. However, due to the challenges of collecting effective training data and learning efficiently, most grasping algorithms today are limited to top-down movements and open-loop execution. In this work, we propose a new low-cost hardware interface for collecting grasping demonstrations by people in diverse environments. Leveraging this data, we show that it is possible to train a robust end-to-end 6DoF closed-loop grasping model with r…
Framework and Multi-modal Dataset for Roadwork Zone Detection and Geo-localization
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences（应用科学英戈尔施塔特技术大学创新移动研究所）；Fraunhofer Institute for Transportation and Infrastructure Systems IVI（弗劳恩霍夫交通与基础设施系统研究所IVI）；Technical University of Munich（慕尼黑工业大学）
