Position sensitive detectors (PSDs) offer possibility to track single active marker's two (or three) degrees of freedom (DoF) position with a high accuracy, while having a fast response time with high update frequency and low latency, all using a very simple signal processing circuit. However they are not particularly suitable for 6-DoF object pose tracking system due to lack of orientation measurement, limited tracking range, and sensitivity to environmental variation. We propose a novel 6-DoF pose tracking system for a rigid object tracking requiring a single active marker. The proposed system uses a stereo-based PSD pair and multiple Inertial Measurement Units (IMUs). This is done based on a practical approach to identify and control the power of Infrared-Light Emitting Diode (IR-LED) active markers, with an aim to increase the tracking work space and reduce the power consumption. Our proposed tracking system is validated with three different work space sizes and for static and dynamic positional accuracy using robotic arm manipulator with three different dynamic motion patterns. The results show that the static position root-mean-square (RMS) error is 0.6mm. The dynamic position RMS error is 0.7-0.9mm. The orientation RMS error is between 0.04 and 0.9 degree at varied dynamic motion. Overall, our proposed tracking system is capable of tracking a rigid object pose with sub-millimeter accuracy at the mid range of the work space and sub-degree accuracy for all work space under a lab setting.
6-DoF Object Tracking with Event-based Optical Flow and Frames
Humanoid Sensing and Perception, Istituto Italiano di Tecnologia（人形感知与感知，意大利技术研究所）；Event-driven Perception for Robotics, Istituto Italiano di Tecnologia（机器人驱动感知，意大利技术研究所）；University of Genoa（热那亚大学）
Université Laval（拉瓦尔大学）；Thales Digital Solutions（泰雷兹数字解决方案）；Inria（法国国家信息与自动化研究所）
Augmented reality devices require multiple sensors to perform various tasks such as localization and tracking. Currently, popular cameras are mostly frame-based (e.g. RGB and Depth) which impose a high data bandwidth and power usage. With the necessity for low power and more responsive augmented reality systems, using solely frame-based sensors imposes limits to the various algorithms that needs high frequency data from the environement. As such, event-based sensors have become increasingly popular due to their low power, bandwidth and latency, as well as their very high frequency data acquisi…
Minimal Solvers for Full-DoF Motion Estimation from Asynchronous Differential SfM
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学空天科学学院）；The Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）
This paper presents a method to control a manipulator system grasping a rigid-body payload so that the motion of the combined system in consequence of externally applied forces to be the same as another free-floating rigid-body (with different inertial properties). This allows zero-g emulation of a scaled spacecraft prototype under the test in a 1-g laboratory environment. The controller consisting of motion feedback and force/moment feedback adjusts the motion of the test spacecraft so as to match that of the flight spacecraft, even if the latter has flexible appendages (such as solar panels)…
As neuromorphic technology is maturing, its application to robotics and autonomous vehicle systems has become an area of active research. In particular, event cameras have emerged as a compelling alternative to frame-based cameras in low-power and latency-demanding applications. To enable event cameras to operate alongside staple sensors like lidar in perception tasks, we propose a direct, temporally-decoupled extrinsic calibration method between event cameras and lidars. The high dynamic range, high temporal resolution, and low-latency operation of event cameras are exploited to directly regi…
Tensegrity robots, which are composed of compressive elements (rods) and flexible tensile elements (e.g., cables), have a variety of advantages, including flexibility, low weight, and resistance to mechanical impact. Nevertheless, the hybrid soft-rigid nature of these robots also complicates the ability to localize and track their state. This work aims to address what has been recognized as a grand challenge in this domain, i.e., the state estimation of tensegrity robots through a markerless, vision-based method, as well as novel, onboard sensors that can measure the length of the robot's cabl…
National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；University of Tsukuba（筑波大学）
In many robotic applications, the environment setting in which the 6-DoF pose estimation of a known, rigid object and its subsequent grasping is to be performed, remains nearly unchanging and might even be known to the robot in advance. In this paper, we refer to this problem as instance-specific pose estimation: the robot is expected to estimate the pose with a high degree of accuracy in only a limited set of familiar scenarios. Minor changes in the scene, including variations in lighting conditions and background appearance, are acceptable but drastic alterations are not anticipated. To this…
Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an…
In this paper we propose a method that estimates the $SE(3)$ continuous trajectories (orientation and translation) of the dynamic rigid objects present in a scene, from multiple RGB-D views. Specifically, we fit the object trajectories to cumulative B-Splines curves, which allow us to interpolate, at any intermediate time stamp, not only their poses but also their linear and angular velocities and accelerations. Additionally, we derive in this work the analytical $SE(3)$ Jacobians needed by the optimization, being applicable to any other approach that uses this type of curves. To the best of o…
Technische Universität München（慕尼黑工业大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
The outbreak of novel coronavirus pneumonia (COVID-19) has caused mortality and morbidity worldwide. Oropharyngeal-swab (OP-swab) sampling is widely used for the diagnosis of COVID-19 in the world. To avoid the clinical staff from being affected by the virus, we developed a 9-degree-of-freedom (DOF) rigid-flexible coupling (RFC) robot to assist the COVID-19 OP-swab sampling. This robot is composed of a visual system, UR5 robot arm, micro-pneumatic actuator and force-sensing system. The robot is expected to reduce risk and free up the clinical staff from the long-term repetitive sampling work. …
Hybrid Dynamics Modeling for a Flexible 2-DoF Robotic Arm
Dept. of Mechanical and Manufacturing Eng at the Schulich School of Engineering, University of Calgary, Alberta, Canada（施密特工程学院机械与制造工程系，卡尔加里大学，阿尔伯塔，加拿大）；Schulich School of Engineering at the University of Calgary（卡尔加里大学施密特工程学院）；Intelligent Dynamics and Control Lab（智能动力与控制实验室）；University of Calgary（卡尔加里大学）
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consis…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Notre Dame（圣母大学）
Model-based control for robots has increasingly been dependent on optimization-based methods like Differential Dynamic Programming and iterative LQR (iLQR). These methods can form the basis of Model-Predictive Control (MPC), which is commonly used for controlling legged robots. Computing the partial derivatives of the dynamics is often the most expensive part of these algorithms, regardless of whether analytical methods, Finite Difference, Automatic Differentiation (AD), or Chain-Rule accumulation is used. Since the second-order derivatives of dynamics result in tensor computations, they are o…
Agriculture is currently undergoing a robotics revolution, but robots using wheeled or treads suffer from known disadvantages: they are unable to move over rubble and steep or loose ground, and they trample continuous strips of land thereby reducing the viable crop area. Legged robots offer an alternative, but existing commercial legged robots are complex, expensive, and hard to maintain. We propose the use of multilegged robots using low-degree-of-freedom (low-DoF) legs and demonstrate our approach with a lawn pest control task: picking dandelions using our inexpensive and easy to fabricate B…
Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera
ShanghaiTech University（上海科技大学）；ShanghaiTech Automation and Robotics (STAR) Center（上海科技大学自动化与机器人中心）；School of Information Science and Technology（信息科学与技术学院）；WLSA Shanghai Academy（WLSA上海研究院）
Université Libre de Bruxelles（布鲁塞尔自由大学）；Entreprises Jacques Delens S.A.（雅克·德朗斯企业股份有限公司）
Automation of cranes can have a direct impact on the productivity of construction projects. In this paper, we focus on the control of one of the most used cranes, the boom crane. Tower cranes and overhead cranes have been widely studied in the literature, whereas the control of boom cranes has been investigated only by a few works. Typically, these works make use of simple models making use of a large number of simplifying assumptions (e.g. fixed length cable, assuming certain dynamics are uncoupled, etc.) A first result of this paper is to present a fairly complete nonlinear dynamic model of …
Many algorithms for control, optimization and estimation in robotics depend on derivatives of the underlying system dynamics, e.g. to compute linearizations, sensitivities or gradient directions. However, we show that when dealing with Rigid Body Dynamics, these derivatives are difficult to derive analytically and to implement efficiently. To overcome this issue, we extend the modelling tool `RobCoGen' to be compatible with Automatic Differentiation. Additionally, we propose how to automatically obtain the derivatives and generate highly efficient source code. We highlight the flexibility and …
Real-time 6-DoF Pose Estimation by an Event-based Camera using Active LED Markers
Automation and Control Institute (ACIN), TU Wien（维也纳工业大学自动化与控制研究所（ACIN））；Austrian Institute of Technology (AIT)（奥地利技术研究院（AIT））；Prophesee GmbH（Prophesee有限公司）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Notre Dame（圣母大学）
Optimization-based robot control strategies often rely on first-order dynamics approximation methods, as in iLQR. Using second-order approximations of the dynamics is expensive due to the costly second-order partial derivatives of the dynamics with respect to the state and control. Current approaches for calculating these derivatives typically use automatic differentiation (AD) and chain-rule accumulation or finite-difference. In this paper, for the first time, we present analytical expressions for the second-order partial derivatives of inverse dynamics for open-chain rigid-body systems with …
University of Edinburgh（爱丁堡大学）；University of Birmingham（伯明翰大学）；Technical University Braunschweig（布伦瑞克工业大学）
We propose a method for dual-arm manipulation of rigid objects, subject to external disturbance. The problem is formulated as a Cartesian impedance controller within a projected inverse dynamics framework. We use the constrained component of the controller to enforce contact and the unconstrained controller to accomplish the task with a desired 6-DOF impedance behaviour. Furthermore, the proposed method optimises the torque required to maintain contact, subject to unknown disturbances, and can do so without direct measurement of external force. The techniques are evaluated on a single-arm wipi…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Notre Dame（圣母大学）
An essential need for many model-based robot control algorithms is the ability to quickly and accurately compute partial derivatives of the equations of motion. State of the art approaches to this problem often use analytical methods based on the chain rule applied to existing dynamics algorithms. Although these methods are an improvement over finite differences in terms of accuracy, they are not always the most efficient. In this paper, we contribute new closed-form expressions for the first-order partial derivatives of inverse dynamics, leading to a recursive algorithm. The algorithm is benc…
This paper addresses the long-standing open problem of observability of mass and inertia plant parameters in the adaptive identification (AID) of second-order nonlinear models of 6 degree-of-freedom rigid-body dynamical systems subject to externally applied forces and moments. Although stable methods for AID of plant parameters for this class of systems, as well numerous approaches to stable model-based direct adaptive trajectory-tracking control of such systems, have been reported, these studies have been unable to prove analytically that the adaptive parameter estimates converge to the true …
Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究所（Skoltech））
The projection mapping systems on the human face is limited by the latency and the movement of the users. The area of the projection is restricted by the position of the projectors and the cameras. We are introducing MaskBot, a real-time projection mapping system operated by a 6 Degrees of Freedom (DoF) collaborative robot. The collaborative robot locates the projector and camera in normal position to the face of the user to increase the projection area and to reduce the latency of the system. A webcam is used to detect the face and to sense the robot-user distance to modify the projection siz…
ArtiMinds Robotics（ArtiMinds机器人公司）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
This paper introduces an efficient procedure to localize user-defined points on the surface of deformable objects and track their positions in 3D space over time. To cope with a deformable object's infinite number of DOF, we propose a discretized deformation field, which is estimated during runtime using a multi-step non-linear solver pipeline. The resulting high-dimensional energy minimization problem describes the deviation between an offline-defined reference model and a pre-processed camera image. An additional regularization term allows for assumptions about the object's hidden areas and …
