This work addresses the issue of motion compensation and pattern tracking in event camera data. An event camera generates asynchronous streams of events triggered independently by each of the pixels upon changes in the observed intensity. Providing great advantages in low-light and rapid-motion scenarios, such unconventional data present significant research challenges as traditional vision algorithms are not directly applicable to this sensing modality. The proposed method decomposes the tracking problem into a local SE(2) motion-compensation step followed by a homography registration of small motion-compensated event batches. The first component relies on Gaussian Process (GP) theory to model the continuous occupancy field of the events in the image plane and embed the camera trajectory in the covariance kernel function. In doing so, estimating the trajectory is done similarly to GP hyperparameter learning by maximising the log marginal likelihood of the data. The continuous occupancy fields are turned into distance fields and used as templates for homography-based registration. By benchmarking the proposed method against other state-of-the-art techniques, we show that our open-source implementation performs high-accuracy motion compensation and produces high-quality tracks in real-world scenarios.
Istituto Italiano di Tecnologia（意大利技术研究院）；Politecnico di Milano（米兰理工大学）
Simulation tools are essential for robotics research, especially for those domains in which safety is crucial, such as Human-Robot Collaboration (HRC). However, it is challenging to simulate human behaviors, and existing robotics simulators do not integrate functional human models. This work presents Open-VICO, an open-source toolkit to integrate virtual human models in Gazebo focusing on vision-based human tracking. In particular, Open-VICO allows to combine in the same simulation environment realistic human kinematic models, multi-camera vision setups, and human-tracking techniques along wit…
FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
Department of Electrical and Computer Engineering, University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校电子与计算机工程系）；Toyota Motor North America R&D（丰田北美研发）
Unmanned aerial vehicles (UAV) have been widely used in various fields, and their invasion of security and privacy has aroused social concern. Several detection and tracking systems for UAVs have been introduced in recent years, but most of them are based on radio frequency, radar, and other media. We assume that the field of computer vision is mature enough to detect and track invading UAVs. Thus we propose a visible light mode dataset called Dalian University of Technology Anti-UAV dataset, DUT Anti-UAV for short. It contains a detection dataset with a total of 10,000 images and a tracking d…
School of Software Technology, Dalian University of Technology（大连理工大学软件学院）；School of Mechanical Engineering, Dalian University of Technology（大连理工大学机械工程学院）；School of Mathematics and Statistics, Xi'an Jiaotong University（西安交通大学数学与统计学学院）；Macao Institute of Systems Engineering, Macau University of Science and Technology（澳门科学理工学院系统工程研究所）
School of Automation, Chongqing University of Posts and Telecommunications（重庆邮电大学自动化学院）
Tracking by detection paradigm is one of the most popular object tracking methods. However, it is very dependent on the performance of the detector. When the detector has a behavior of missing detection, the tracking result will be directly affected. In this paper, we analyze the phenomenon of the lost tracking object in real-time tracking model on MOT2020 dataset. Based on simple and traditional methods, we propose a compensation tracker to further alleviate the lost tracking problem caused by missing detection. It consists of a motion compensation module and an object selection module. The p…
Alpen-Adria-Universität Klagenfurt（阿尔卑斯-亚得里亚大学克拉根福分校）；University of Sarajevo（萨拉热窝大学）
Object tracking is one of the most important and fundamental disciplines of Computer Vision. Many Computer Vision applications require specific object tracking capabilities, including autonomous and smart vehicles, video surveillance, medical treatments, and many others. The OpenCV as one of the most popular libraries for Computer Vision includes several hundred Computer Vision algorithms. Object tracking tasks in the library can be roughly clustered in single and multiple object trackers. The library is widely used for real-time applications, but there are a lot of unanswered questions such a…
Artificial Microsaccade Compensation: Stable Vision for an Ornithopter
Department of Computer Science, University of Maryland, College Park（计算机科学系，马里兰大学，学院公园）；Faculty of Aerospace Engineering, Delft University of Technology（航空航天工程系，代尔夫特理工大学）；Institute for Advanced Computer Studies, University of Maryland, College Park（高级计算机研究 institute，马里兰大学，学院公园）
China-UAE Belt and Road Joint Laboratory on Intelligent Unmanned Systems, School of Aerospace Engineering, Beijing Institute of Technology（中国-阿联酋“一带一路”智能无人系统联合实验室，北京理工大学宇航学院）
Hamburg University of Technology（汉堡工业大学）；eemagine Medical Imaging Solutions GmbH（eemagine医学影像解决方案有限公司）
Head pose estimation and tracking is useful in variety of medical applications. With the advent of RGBD cameras like Kinect, it has become feasible to do markerless tracking by estimating the head pose directly from the point clouds. One specific medical application is robot assisted transcranial magnetic stimulation (TMS) where any patient motion is compensated with the help of a robot. For increased patient comfort, it is important to track the head without markers. In this regard, we address the head pose estimation problem using two different approaches. In the first approach, we build upo…
In order to improve the precision of multi-robot SLAM multi-view target tracking process, a improved multi-robot SLAM multi-view target tracking algorithm based on panoramic vision in irregular environment was put forward, adding an correction factor to renew the existing Extended Kalman Filter (EKF) model, obtaining new coordinates X and Y after twice iterations. The paper has been accepted by Computing and Visualization in Science and this is a simplified version.
In this paper, we present a novel vision-based framework for tracking dynamic objects using guidance laws based on a rendezvous cone approach. These guidance laws enable an unmanned aircraft system equipped with a monocular camera to continuously follow a moving object within the sensor's field of view. We identify and classify feature point estimators for managing the occurrence of occlusions during the tracking process in an exclusive manner. Furthermore, we develop an open-source simulation environment and perform a series of simulations to show the efficacy of our methods.
Tongji University（同济大学）；the University of Hong Kong（香港大学）
Vision-based object tracking has boosted extensive autonomous applications for unmanned aerial vehicles (UAVs). However, the dynamic changes in flight maneuver and viewpoint encountered in UAV tracking pose significant difficulties, e.g. , aspect ratio change, and scale variation. The conventional cross-correlation operation, while commonly used, has limitations in effectively capturing perceptual similarity and incorporates extraneous background information. To mitigate these limitations, this work presents a novel saliency-guided dynamic vision Transformer (SGDViT) for UAV tracking. The prop…
Tongji University（同济大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）
Although the manipulating of the unmanned aerial manipulator (UAM) has been widely studied, vision-based UAM approaching, which is crucial to the subsequent manipulating, generally lacks effective design. The key to the visual UAM approaching lies in object tracking, while current UAM tracking typically relies on costly model-based methods. Besides, UAM approaching often confronts more severe object scale variation issues, which makes it inappropriate to directly employ state-of-the-art model-free Siamese-based methods from the object tracking field. To address the above problems, this work pr…
The field of robotic vision is developing rapidly. Robots can react intelligently and provide assistance to user activities through sentient computing. Since industrial applications pose complex requirements that cannot be handled by humans, an efficient low cost and robust technique is required for the tracking of mobile industrial robots. The existing sensor based techniques for mobile robot tracking are expensive and complex to deploy, configure and maintain. Also some of them demand dedicated and often expensive hardware. This paper presents a low cost vision based technique called Hybrid …
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Singapore Technologies Dynamics Pte Ltd（新加坡科技动力私人有限公司）
Outdoor vision-based systems suffer from atmospheric turbulences, and rain is one of the worst factors for vision degradation. Current rain removal methods show limitations either for complex dynamic scenes, or under torrential rain with opaque occlusions. We propose a novel derain framework which applies superpixel (SP) segmentation to decompose the scene into depth consistent units. Alignment of scene contents are done at the SP level, which proves to be robust against rain occlusion interferences and fast camera motion. Two alignment output tensors, i.e., optimal temporal match tensor and s…
In many computer vision domains, the input images must conform with the pinhole camera model, where straight lines in the real world are projected as straight lines in the image. Performing computer vision tasks on live sports broadcast footage imposes challenging requirements where the algorithms cannot rely on a specific calibration pattern must be able to cope with unknown and uncalibrated cameras, radial distortion originating from complex television lenses, few visual clues to compensate distortion by, and the necessity for real-time performance. We present a novel method for single-image…
Real-time tracking is an important problem in computer vision in which most methods are based on the conventional cameras. Neuromorphic vision is a concept defined by incorporating neuromorphic vision sensors such as silicon retinas in vision processing system. With the development of the silicon technology, asynchronous event-based silicon retinas that mimic neuro-biological architectures has been developed in recent years. In this work, we combine the vision tracking algorithm of computer vision with the information encoding mechanism of event-based sensors which is inspired from the neural …
Event cameras are novel vision sensors that output pixel-level brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them Focus Loss Functions since they have strong connections wit…
To address the SMC'17 data challenge -- "Data mining atomically resolved images for material properties", we first used the classic "blob detection" algorithms developed in computer vision to identify all atom centers in each STEM image frame. With the help of nearest neighbor analysis, we then found and labeled every atom center common to all the STEM frames and tracked their movements through the given time interval for both Molybdenum or Selenium atoms.
A Survey of Fish Tracking Techniques Based on Computer Vision
College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）；National Innovation Center for Digital Fishery, Ministry of Agriculture and Rural Affairs（农业农村部数字渔业创新中心）；College of Engineering, China Agricultural University（中国农业大学工学院）
The paper focuses on the problem of vision-based obstacle detection and tracking for unmanned aerial vehicle navigation. A real-time object localization and tracking strategy from monocular image sequences is developed by effectively integrating the object detection and tracking into a dynamic Kalman model. At the detection stage, the object of interest is automatically detected and localized from a saliency map computed via the image background connectivity cue at each frame; at the tracking stage, a Kalman filter is employed to provide a coarse prediction of the object state, which is furthe…
This paper describes the kinematics of the motion tracking of a rigid body using video recording. The novelty of the paper is on the adaptation of the methods and nomenclature used in Computer Vision to those used in Multibody System Dynamics. That way, the equations presented here can be used, for example, for inverse-dynamics multibody simulations driven by the motion tracking of selected bodies. This paper also adapts the well-known Zhang calibration method to the presented nomenclature.
Global motion compensation (GMC) removes the impact of camera motion and creates a video in which the background appears static over the progression of time. Various vision problems, such as human activity recognition, background reconstruction, and multi-object tracking can benefit from GMC. Existing GMC algorithms rely on sequentially processing consecutive frames, by estimating the transformation mapping the two frames, and obtaining a composite transformation to a global motion compensated coordinate. Sequential GMC suffers from temporal drift of frames from the accurate global coordinate,…
University of Udine（乌迪内大学）；University of Catania（卡塔尼亚大学）
Understanding human-object interactions is fundamental in First Person Vision (FPV). Tracking algorithms which follow the objects manipulated by the camera wearer can provide useful cues to effectively model such interactions. Visual tracking solutions available in the computer vision literature have significantly improved their performance in the last years for a large variety of target objects and tracking scenarios. However, despite a few previous attempts to exploit trackers in FPV applications, a methodical analysis of the performance of state-of-the-art trackers in this domain is still m…
Automatic motion compensation and adjustment of an intraoperative imaging modality's field of view is a common problem during interventions. Optical coherence tomography (OCT) is an imaging modality which is used in interventions due to its high spatial resolution of few micrometers and its temporal resolution of potentially several hundred volumes per second. However, performing motion compensation with OCT is problematic due to its small field of view which might lead to tracked objects being lost quickly. We propose a novel deep learning-based approach that directly learns input parameters …
Freie Universität Berlin（柏林自由大学）；University of Applied Science Kempten（肯普滕应用科学大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Leibniz-Institute of Freshwater Ecology and Inland Fisheries（莱布尼茨淡水生态与内陆渔业研究所）
The study of animal behavior increasingly relies on (semi-) automatic methods for the extraction of relevant behavioral features from video or picture data. To date, several specialized software products exist to detect and track animals' positions in simple (laboratory) environments. Tracking animals in their natural environments, however, often requires substantial customization of the image processing algorithms to the problem-specific image characteristics. Here we introduce BioTracker, an open-source computer vision framework, that provides programmers with core functionalities that are e…
Instituto Nacional de Astrofísica, Óptica y Electrónica（国家天体物理、光学与电子学研究所）；Escuela Politécnica Nacional（国立理工学院）
This paper describes the development of an embedded vision system for detection, location, and tracking of a color object; it makes use of a single 32-bit microprocessor to acquire image data, process, and perform actions according to the interpreted data. The system is intended for applications that need to make use of artificial vision for detection, location and tracking of a color object and its objective is to have achieve at reduced terms of size, power consumption, and cost.
The goal of multi-object tracking (MOT) is detecting and tracking all the objects in a scene, while keeping a unique identifier for each object. In this paper, we present a new robust state-of-the-art tracker, which can combine the advantages of motion and appearance information, along with camera-motion compensation, and a more accurate Kalman filter state vector. Our new trackers BoT-SORT, and BoT-SORT-ReID rank first in the datasets of MOTChallenge [29, 11] on both MOT17 and MOT20 test sets, in terms of all the main MOT metrics: MOTA, IDF1, and HOTA. For MOT17: 80.5 MOTA, 80.2 IDF1, and 65.…
Visual Computing Institute, RWTH Aachen University（亚琛工业大学视觉计算研究所）
The most common paradigm for vision-based multi-object tracking is tracking-by-detection, due to the availability of reliable detectors for several important object categories such as cars and pedestrians. However, future mobile systems will need a capability to cope with rich human-made environments, in which obtaining detectors for every possible object category would be infeasible. In this paper, we propose a model-free multi-object tracking approach that uses a category-agnostic image segmentation method to track objects. We present an efficient segmentation mask-based tracker which associ…
Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间应用工程与技术中心）；University of Chinese Academy of Sciences（中国科学院大学）
University of Udine（乌迪内大学）；University of Catania（卡塔尼亚大学）
Understanding human-object interactions is fundamental in First Person Vision (FPV). Tracking algorithms which follow the objects manipulated by the camera wearer can provide useful cues to effectively model such interactions. Despite a few previous attempts to exploit trackers in FPV applications, a methodical analysis of the performance of state-of-the-art visual trackers in this domain is still missing. In this short paper, we provide a recap of the first systematic study of object tracking in FPV. Our work extensively analyses the performance of recent and baseline FPV trackers with respec…
Mobile Augmented Reality (MAR) is becoming an important cyber-physical system application given the ubiquitous availability of mobile phones. With the need to operate in unprepared environments, accurate and robust registration and tracking has become an important research problem to solve. In fact, when MAR is used for tele-interactive applications involving large distances, say from an accident site to insurance office, tracking at both the ends is desirable and further it is essential to appropriately fuse inertial and vision sensors data. In this paper, we present results and discuss some …
Beijing Jiaotong University（北京交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of North Texas（北得克萨斯大学）
Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning …
Hong Kong University of Science and Technology（香港科技大学）
We propose a stereo vision-based approach for tracking the camera ego-motion and 3D semantic objects in dynamic autonomous driving scenarios. Instead of directly regressing the 3D bounding box using end-to-end approaches, we propose to use the easy-to-labeled 2D detection and discrete viewpoint classification together with a light-weight semantic inference method to obtain rough 3D object measurements. Based on the object-aware-aided camera pose tracking which is robust in dynamic environments, in combination with our novel dynamic object bundle adjustment (BA) approach to fuse temporal sparse…
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
In this paper, we present EBBIOT-a novel paradigm for object tracking using stationary neuromorphic vision sensors in low-power sensor nodes for the Internet of Video Things (IoVT). Different from fully event based tracking or fully frame based approaches, we propose a mixed approach where we create event-based binary images (EBBI) that can use memory efficient noise filtering algorithms. We exploit the motion triggering aspect of neuromorphic sensors to generate region proposals based on event density counts with >1000X less memory and computes compared to frame based approaches. We also prop…
With a video data source, such as multispectral video acquired during administration of fluorescent tracers, extraction of time-resolved data typically requires the compensation of motion. While this can be done manually, which is arduous, or using off-the-shelf object tracking software, which often yields unsatisfactory performance, we present an algorithm which is simple and performant. Most importantly, we provide an open-source implementation, with an easy-to-use interface for researchers not inclined to write their own code, as well as Python modules that can be used programmatically.
University of Udine（乌迪内大学）；University of Catania（卡塔尼亚大学）
The understanding of human-object interactions is fundamental in First Person Vision (FPV). Visual tracking algorithms which follow the objects manipulated by the camera wearer can provide useful information to effectively model such interactions. In the last years, the computer vision community has significantly improved the performance of tracking algorithms for a large variety of target objects and scenarios. Despite a few previous attempts to exploit trackers in the FPV domain, a methodical analysis of the performance of state-of-the-art trackers is still missing. This research gap raises …
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Haifa（海法大学）；University of São Paulo（圣保罗大学）；Swedish University of Agricultural Sciences（瑞典农业科学大学）；Universiteit Utrecht（乌得勒支大学）；University of Glasgow（格拉斯哥大学）；Silo AI；Boğaziçi University（博阿齐奇大学）
Advances in animal motion tracking and pose recognition have been a game changer in the study of animal behavior. Recently, an increasing number of works go 'deeper' than tracking, and address automated recognition of animals' internal states such as emotions and pain with the aim of improving animal welfare, making this a timely moment for a systematization of the field. This paper provides a comprehensive survey of computer vision-based research on recognition of affective states and pain in animals, addressing both facial and bodily behavior analysis. We summarize the efforts that have been…
