Active Search and Tracking for search and rescue missions or collaborative mobile robotics relies on the actuation of a sensing platform to detect and localize a target. In this paper we focus on visually detecting a radio-emitting target with an aerial robot equipped with a radio receiver and a camera. Visual-based tracking provides high accuracy, but the directionality of the sensing domain may require long search times before detecting the target. Conversely, radio signals have larger coverage, but lower tracking accuracy. Thus, we design a Recursive Bayesian Estimation scheme that uses camera observations to refine radio measurements. To regulate the camera pose, we design an optimal controller whose cost function is built upon a probabilistic map. Theoretical results support the proposed algorithm, while numerical analyses show higher robustness and efficiency with respect to visual and radio-only baselines.
This paper proposes a novel multi-target tracking (MTT) algorithm for scenarios with arbitrary numbers of measurements per target. We propose the variational probabilistic multi-hypothesis tracking (VPMHT) algorithm based on the variational Bayesian expectation-maximisation (VBEM) algorithm to resolve the MTT problem in the classic PMHT algorithm. With the introduction of variational inference, the proposed VPMHT handles track-loss much better than the conventional probabilistic multi-hypothesis tracking (PMHT) while preserving a similar or even better tracking accuracy. Extensive numerical si…
Multi-modal Probabilistic Active Sensing (MMPAS) uses sensor fusion and probabilistic models to control the perception process of robotic sensing platforms. MMPAS is successfully employed in environmental exploration, collaborative mobile robotics, and target tracking, being fostered by the high performance guarantees on autonomous perception. In this context, we propose a bi-Radio-Visual PAS scheme to solve the transmitter discovery problem. Specifically, we firstly exploit the correlation between radio and visual measurements to learn a target detection model in a self-supervised manner. The…
The problem of multi-robot target tracking asks for actively planning the joint motion of robots to track targets. In this paper, we focus on such target tracking problems in adversarial environments, where attacks or failures may deactivate robots' sensors and communications. In contrast to the previous works that consider no attacks or sensing attacks only, we formalize the first robust multi-robot tracking framework that accounts for any fixed numbers of worst-case sensing and communication attacks. To secure against such attacks, we design the first robust planning algorithm, named Robust …
In this paper, we study integrated estimation and control of soft robots. A significant challenge in deploying closed loop controllers is reliable proprioception via integrated sensing in soft robots. Despite the considerable advances accomplished in fabrication, modelling, and model-based control of soft robots, integrated sensing and estimation is still in its infancy. To that end, this paper introduces a new method of estimating the degree of curvature of a soft robot using a stretchable sensing skin. The skin is a spray-coated piezoresistive sensing layer on a latex membrane. The mapping f…
Resilient Multi-Robot Target Tracking with Sensing and Communication Danger Zones
Department of Electrical and Computer Engineering, Drexel University（电子与计算机工程系，德雷塞尔大学）；GRASP Lab, University of Pennsylvania（GRASP实验室，宾夕法尼亚大学）；Institute for Robotics and Intelligent Machines, Georgia Institute of Technology（机器人与智能机器研究所，佐治亚理工学院）；Department of Computer Science, University of Southern California（计算机科学系，南加州大学）
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究院）
Multi-object tracking is an important ability for an autonomous vehicle to safely navigate a traffic scene. Current state-of-the-art follows the tracking-by-detection paradigm where existing tracks are associated with detected objects through some distance metric. The key challenges to increase tracking accuracy lie in data association and track life cycle management. We propose a probabilistic, multi-modal, multi-object tracking system consisting of different trainable modules to provide robust and data-driven tracking results. First, we learn how to fuse features from 2D images and 3D LiDAR …
3D multi-object tracking is a key module in autonomous driving applications that provides a reliable dynamic representation of the world to the planning module. In this paper, we present our on-line tracking method, which made the first place in the NuScenes Tracking Challenge, held at the AI Driving Olympics Workshop at NeurIPS 2019. Our method estimates the object states by adopting a Kalman Filter. We initialize the state covariance as well as the process and observation noise covariance with statistics from the training set. We also use the stochastic information from the Kalman Filter in …
Precisely tracking uncertainties is crucial for robots to successfully and safely operate in unstructured and dynamic environments. We present a probabilistic framework to precisely keep track of uncertainties throughout the entire manipulation process. In agreement with common manipulation pipelines, we decompose the process into two subsequent stages, namely perception and physical interaction. Each stage is associated with different sources and types of uncertainties, requiring different techniques. We discuss which representation of uncertainties is the most appropriate for each stage (e.g…
University of Zagreb, Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）
This paper studies the interplay between kinematics (position and velocity) and appearance cues for establishing correspondences in multi-target pedestrian tracking. We investigate tracking-by-detection approaches based on a deep learning detector, joint integrated probabilistic data association (JIPDA), and appearance-based tracking of deep correspondence embeddings. We first addressed the fixed-camera setup by fine-tuning a convolutional detector for accurate pedestrian detection and combining it with kinematic-only JIPDA. The resulting submission ranked first on the 3DMOT2015 benchmark. How…
Dense image alignment from RGB-D images remains a critical issue for real-world applications, especially under challenging lighting conditions and in a wide baseline setting. In this paper, we propose a new framework to learn a pixel-wise deep feature map and a deep feature-metric uncertainty map predicted by a Convolutional Neural Network (CNN), which together formulate a deep probabilistic feature-metric residual of the two-view constraint that can be minimised using Gauss-Newton in a coarse-to-fine optimisation framework. Furthermore, our network predicts a deep initial pose for faster and …
Max-Planck-Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Southern California（南加州大学）
We address the problem of tracking the 6-DoF pose of an object while it is being manipulated by a human or a robot. We use a dynamic Bayesian network to perform inference and compute a posterior distribution over the current object pose. Depending on whether a robot or a human manipulates the object, we employ a process model with or without knowledge of control inputs. Observations are obtained from a range camera. As opposed to previous object tracking methods, we explicitly model self-occlusions and occlusions from the environment, e.g, the human or robotic hand. This leads to a strongly no…
Given a mapped environment, we formulate the problem of visually tracking and following an evader using a probabilistic framework. In this work, we consider a non-holonomic robot with a limited visibility depth sensor in an indoor environment with obstacles. The mobile robot that follows the target is considered a pursuer and the agent being followed is considered an evader. We propose a probabilistic framework for both the pursuer and evader to achieve their conflicting goals. We introduce a smart evader that has information about the location of the pursuer. The goal of this variant of the e…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Southern California（南加州大学）
We propose a probabilistic filtering method which fuses joint measurements with depth images to yield a precise, real-time estimate of the end-effector pose in the camera frame. This avoids the need for frame transformations when using it in combination with visual object tracking methods. Precision is achieved by modeling and correcting biases in the joint measurements as well as inaccuracies in the robot model, such as poor extrinsic camera calibration. We make our method computationally efficient through a principled combination of Kalman filtering of the joint measurements and asynchronous…
This paper addresses the topic of semantic world modeling by conjoining probabilistic reasoning and object anchoring. The proposed approach uses a so-called bottom-up object anchoring method that relies on the rich continuous data from perceptual sensor data. A novel anchoring matching function method learns to maintain object entities in space and time and is validated using a large set of trained humanly annotated ground truth data of real-world objects. For more complex scenarios, a high-level probabilistic object tracker has been integrated with the anchoring framework and handles the trac…
This paper addresses the localization of contacts of an unknown grasped rigid object with its environment, i.e., extrinsic to the robot. We explore the key role that distributed tactile sensing plays in localizing contacts external to the robot, in contrast to the role that aggregated force/torque measurements play in localizing contacts on the robot. When in contact with the environment, an object will move in accordance with the kinematic and possibly frictional constraints imposed by that contact. Small motions of the object, which are observable with tactile sensors, indirectly encode thos…
University at Buffalo（布法罗大学）；Clemson University（克莱姆森大学）
This paper proposes a multi-sensor based approach to detect, track, and localize a quadcopter unmanned aerial vehicle (UAV). Specifically, a pipeline is developed to process monocular RGB and thermal video (captured from a fixed platform) to detect and track the UAV in our FoV. Subsequently, a 2D planar lidar is used to allow conversion of pixel data to actual distance measurements, and thereby enable localization of the UAV in global coordinates. The monocular data is processed through a deep learning-based object detection method that computes an initial bounding box for the UAV. The thermal…
We address the problem of maintaining resource availability in a networked multi-robot system performing distributed target tracking. In our model, robots are equipped with sensing and computational resources enabling them to track a target's position using a Distributed Kalman Filter (DKF). We use the trace of each robot's sensor measurement noise covariance matrix as a measure of sensing quality. When a robot's sensing quality deteriorates, the system's communication graph is modified by adding edges such that the robot with deteriorating sensor quality may share information with other robot…
We present Autonomous Rssi based RElative poSitioning and Tracking (ARREST), a new robotic sensing system for tracking and following a moving, RF-emitting object, which we refer to as the Leader, solely based on signal strength information. This kind of system can expand the horizon of autonomous mobile tracking and distributed robotics into many scenarios with limited visibility such as nighttime, dense forests, and cluttered environments. Our proposed tracking agent, which we refer to as the TrackBot, uses a single rotating, off-the-shelf, directional antenna, novel angle and relative speed …
Institute for Transport Studies (ITS), University of Leeds（利兹大学交通研究所）；Lincoln Centre for Autonomous Systems, University of Lincoln（林肯大学林肯自主系统中心）；Institute of Engineering Sciences, De Montfort University（德蒙福特大学工程科学研究所）；School of Mechanical Engineering, Nat. Tech. University of Athens（雅典国家技术大学机械工程学院）；Robert Bosch GmbH（罗伯特·博世有限公司）；Ibex Automation Ltd（伊贝克斯自动化有限公司）
Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behaviour as well as detecting and tracking them. This narrative review article is Part I of a pair, together surveying the current technology stack involved in this process, o…
The Integrated Probabilistic Data Association Filter (IPDAF) is a target tracking algorithm based on the Probabilistic Data Association Filter that calculates a statistical measure that indicates if an estimated representation of the target properly represents the target or is generated from non-target-originated measurements. The main contribution of this paper is to adapt the IPDAF to constant velocity target models that evolve on connected, unimodular Lie groups, and where the measurements are also defined on a Lie group. We present an example where the methods developed in the paper are ap…
Probabilistic Trajectory GOSPA: A Metric for Uncertainty-Aware Multi-Object Tracking Performance Evaluation
Department of Automation and Intelligent Sensing, Shanghai Jiao Tong University（自动化与智能感知系，上海交通大学）；Information Processing and Telecommunications Center, Universidad Politécnica de Madrid（信息处理与电信中心，马德里理工大学）；Department of Mathematics, KTH Royal Institute of Technology（数学系，皇家理工学院）；Department of Electrical Engineering, Chalmers University of Technology（电气工程系，查尔姆斯理工大学）
Observability-driven Assignment of Heterogeneous Sensors for Multi-Target Tracking
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, School of Automation and Intelligent Manufacturing, Southern University of Science and Technology (SUSTech)（广东省全驱动系统控制理论与技术重点实验室，自动化与智能制造学院，南方科技大学）
Optimal control is often used in robotics for planning a trajectory to achieve some desired behavior, as expressed by the cost function. Most works in optimal control focus on finding a single optimal trajectory, which is then typically tracked by another controller. In this work, we instead consider trajectory distribution as the solution of an optimal control problem, resulting in better tracking performance and a more stable controller. A Gaussian distribution is first obtained from an iterative Linear Quadratic Regulator (iLQR) solver. A short horizon Model Predictive Control (MPC) is then…
We present Neural Contact Fields, a method that brings together neural fields and tactile sensing to address the problem of tracking extrinsic contact between object and environment. Knowing where the external contact occurs is a first step towards methods that can actively control it in facilitating downstream manipulation tasks. Prior work for localizing environmental contacts typically assume a contact type (e.g. point or line), does not capture contact/no-contact transitions, and only works with basic geometric-shaped objects. Neural Contact Fields are the first method that can track arbit…
Retrieving an object from cluttered spaces suchas cupboards, refrigerators, or bins requires tracking objects with limited or no visual sensing. In these scenarios, contact feedback is necessary to estimate the pose of the objects, yet the objects are movable while their shapes and number may be unknown, making the association of contacts with objects extremely difficult. While previous work has focused on multi-target tracking, the assumptions therein prohibit using prior methods given only the contact-sensing modality. Instead, this paper proposes the method Soft Tracking Using Contacts for …
Dressing is a fundamental task of everyday living and robots offer an opportunity to assist people with motor impairments. While several robotic systems have explored robot-assisted dressing, few have considered how a robot can manage errors in human pose estimation, or adapt to human motion in real time during dressing assistance. In addition, estimating pose changes due to human motion can be challenging with vision-based techniques since dressing is often intended to visually occlude the body with clothing. We present a method to track a person's pose in real time using capacitive proximity…
In order to enable high-quality decision making and motion planning of intelligent systems such as robotics and autonomous vehicles, accurate probabilistic predictions for surrounding interactive objects is a crucial prerequisite. Although many research studies have been devoted to making predictions on a single entity, it remains an open challenge to forecast future behaviors for multiple interactive agents simultaneously. In this work, we take advantage of the Generative Adversarial Network (GAN) due to its capability of distribution learning and propose a generic multi-agent probabilistic p…
Accurately tracking and predicting behaviors of surrounding objects are key prerequisites for intelligent systems such as autonomous vehicles to achieve safe and high-quality decision making and motion planning. However, there still remain challenges for multi-target tracking due to object number fluctuation and occlusion. To overcome these challenges, we propose a constrained mixture sequential Monte Carlo (CMSMC) method in which a mixture representation is incorporated in the estimated posterior distribution to maintain multi-modality. Multiple targets can be tracked simultaneously within a …
We examine the problem of target tracking whilst simultaneously preserving the target's privacy as epitomized by the robotic panda tracking scenario, which O'Kane introduced at the 2008 Workshop on the Algorithmic Foundations of Robotics in order to elegantly illustrate the utility of ignorance. The present paper reconsiders his formulation and the tracking strategy he proposed, along with its completeness. We explore how the capabilities of the robot and panda affect the feasibility of tracking with a privacy stipulation, uncovering intrinsic limits, no matter the strategy employed. This pape…
Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing
Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Shanghai Qizhi Institute（上海期智研究院）；Department of Mechanical Engineering, Tsinghua University（清华大学机械工程系）；Northwestern Polytechnical University（西北工业大学）；Center on Frontiers of Computing Studies, Peking University（北京大学计算与前沿研究中心）；Shanghai AI Laboratory（上海人工智能实验室）
Successive Pose Estimation and Beam Tracking for mmWave Vehicular Communication Systems
National University of Singapore（国立新加坡大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Southern University of Science and Technology（南方科技大学）；Queen Mary University of London（伦敦女王玛丽大学）；The University of Hong Kong（香港大学）
