Visual-inertial systems rely on precise calibrations of both camera intrinsics and inter-sensor extrinsics, which typically require manually performing complex motions in front of a calibration target. In this work we present a novel approach to obtain favorable trajectories for visual-inertial system calibration, using model-based deep reinforcement learning. Our key contribution is to model the calibration process as a Markov decision process and then use model-based deep reinforcement learning with particle swarm optimization to establish a sequence of calibration trajectories to be performed by a robot arm. Our experiments show that while maintaining similar or shorter path lengths, the trajectories generated by our learned policy result in lower calibration errors compared to random or handcrafted trajectories.
Visual-inertial sensors have a wide range of applications in robotics. However, good performance often requires different sophisticated motion routines to accurately calibrate camera intrinsics and inter-sensor extrinsics. This work presents a novel formulation to learn a motion policy to be executed on a robot arm for automatic data collection for calibrating intrinsics and extrinsics jointly. Our approach models the calibration process compactly using model-free deep reinforcement learning to derive a policy that guides the motions of a robotic arm holding the sensor to efficiently collect m…
Fast Extrinsic Calibration for Multiple Inertial Measurement Units in Visual-Inertial System
Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Magnetometer and inertial sensors are widely used for orientation estimation. Magnetometer usage is often troublesome, as it is prone to be interfered by onboard or ambient magnetic disturbance. The onboard soft-iron material distorts not only the magnetic field, but the magnetometer sensor frame coordinate and the cross-sensor misalignment relative to inertial sensors. It is desirable to conveniently put magnetic and inertial sensors information in a common frame. Existing methods either split the problem into successive intrinsic and cross-sensor calibrations, or rely on stationary accelerom…
University of Delaware（特拉华大学）；Technical University of Munich（慕尼黑工业大学）
In this paper, we study in-depth the problem of online self-calibration for robust and accurate visual-inertial state estimation. In particular, we first perform a complete observability analysis for visual-inertial navigation systems (VINS) with full calibration of sensing parameters, including IMU and camera intrinsics and IMU-camera spatial-temporal extrinsic calibration, along with readout time of rolling shutter (RS) cameras (if used). We investigate different inertial model variants containing IMU intrinsic parameters that encompass most commonly used models for low-cost inertial sensors…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
In this paper, we analyze the observability of the visual-inertial odometry (VIO) using stereo cameras with a velocity-control based kinematic motion model. Previous work shows that in general case the global position and yaw are unobservable in VIO system, additionally the roll and pitch become also unobservable if there is no rotation. We prove that by integrating a planar motion constraint roll and pitch become observable. We also show that the parameters of the motion model are observable.
Autonomous Systems Lab, ETH Zürich（苏黎世联邦理工学院自主系统实验室）；Alibaba Group（阿里巴巴集团）
External effects such as shocks and temperature variations affect the calibration of visual-inertial sensor systems and thus they cannot fully rely on factory calibrations. Re-calibrations performed on short user-collected datasets might yield poor performance since the observability of certain parameters is highly dependent on the motion. Additionally, on resource-constrained systems (e.g mobile phones), full-batch approaches over longer sessions quickly become prohibitively expensive. In this paper, we approach the self-calibration problem by introducing information theoretic metrics to asse…
South China University of Technology（华南理工大学）；City University of Hong Kong（香港城市大学）
In this paper, we present a tightly coupled optimization-based GPS-Visual-Inertial odometry system to solve the trajectory drift of the visual-inertial odometry especially over long-term runs. Visual reprojection residuals, IMU residuals, and GPS measurement residuals are jointly minimized within a local bundle adjustment, in which we apply GPS measurements and IMU preintegration used for the IMU residuals to formulate a novel GPS residual. To improve the efficiency and robustness of the system, we propose a fast reference frames initialization method and an online calibration method for GPS-I…
Environmental conditions and external effects, such as shocks, have a significant impact on the calibration parameters of visual-inertial sensor systems. Thus long-term operation of these systems cannot fully rely on factory calibration. Since the observability of certain parameters is highly dependent on the motion of the device, using short data segments at device initialization may yield poor results. When such systems are additionally subject to energy constraints, it is also infeasible to use full-batch approaches on a big dataset and careful selection of the data is of high importance. I…
Radar-Inertial Odometry with Online Spatio-Temporal Calibration via Continuous-Time IMU Modeling
University of Zagreb Faculty of Electrical Engineering and Computing, Laboratory for Autonomous Systems and Mobile Robotics (LAMOR)（泽格布大学电子工程与计算学院，自主系统与移动机器人实验室）；Calirad d.o.o.（Calirad公司）
Inertial measurement unit (IMU) and odometer have been commonly-used sensors for autonomous land navigation in the global positioning system (GPS)-denied scenarios. This paper systematically proposes a versatile strategy for self-contained land vehicle navigation using the IMU and an odometer. Specifically, the paper proposes a self-calibration and refinement method for IMU/odometer integration that is able to overcome significant variation of the misalignment parameters, which are induced by many inevitable and adverse factors such as load changing, refueling and ambient temperature. An odome…
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Automation, SouthEast University（东南大学自动化学院）
Mobile robot positioning, mapping, and navigation systems generally employ an inertial measurement unit (IMU) to obtain the acceleration and angular velocity of the robot. However, errors in the internal and external parameters of an IMU arising from defective calibration directly affect the accuracy of robot positioning and pose estimation. While this issue has been addressed by the mature internal reference calibration methods available for IMUs, external reference calibration methods between the IMU and the chassis of a mobile robot are lacking. This study addresses this issue by proposing …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We present a method of extrinsic calibration for a system of multiple inertial measurement units (IMUs) that estimates the relative pose of each IMU on a rigid body using only measurements from the IMUs themselves, without the need to prescribe the trajectory. Our method is based on solving a nonlinear least-squares problem that penalizes inconsistency between measurements from pairs of IMUs. We validate our method with experiments both in simulation and in hardware. In particular, we show that it meets or exceeds the performance -- in terms of error, success rate, and computation time -- of a…
The widely-used Extended Kalman Filter (EKF) provides a straightforward recipe to estimate the mean and covariance of the state given all past measurements in a causal and recursive fashion. For a wide variety of applications, the EKF is known to produce accurate estimates of the mean and typically inaccurate estimates of the covariance. For applications in visual inertial localization, we show that inaccuracies in the covariance estimates are \emph{systematic}, i.e. it is possible to learn a nonlinear map from the empirical ground truth to the estimated one. This is demonstrated on both a sta…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
The combination of ultrawideband (UWB) radios and inertial measurement units (IMU) can provide accurate positioning in environments where the Global Positioning System (GPS) service is either unavailable or has unsatisfactory performance. The two sensors, IMU and UWB radio, are often not co-located on a moving system. The UWB radio is typically located at the extremities of the system to ensure reliable communication, whereas the IMUs are located closer to its center of gravity. Furthermore, without hardware or software synchronization, data from heterogeneous sensors can arrive at different t…
Wuhan University（武汉大学）；Wuhan Institute of Quantum Technology（武汉量子技术研究院）；ETH Zürich（苏黎世联邦理工学院）；The Ohio State University（俄亥俄州立大学）
Camera-IMU (Inertial Measurement Unit) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether both camera and IMU intrinsic parameters are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera-IMU system, all intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera, are observable with an unknown landmark. To our kn…
EKF-Based Radar-Inertial Odometry with Online Temporal Calibration
Department of Mechanical Engineering, Ulsan National Institute of Science and Technology (UNIST)（机械工程系，乌山国家科学技术研究院）；Department of Electrical and Electronic Engineering, Imperial College London（电气与电子工程系，伦敦帝国理工学院）
Robust Online Calibration for UWB-Aided Visual-Inertial Navigation with Bias Correction
Department of Electrical and Computer Engineering, George Mason University（乔治·马歇尔大学电气与计算机工程系）；Department of Electrical and Computer Engineering, University of California, Riverside（加州大学河滨分校电气与计算机工程系）；Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville（田纳西大学克林顿分校电气工程与计算机科学系）
Accurate Calibration and Robust LiDAR-Inertial Odometry for Spinning Actuated LiDAR Systems
Engineering Research Center of Low-Altitude Perception and Detection Technology & Intelligent IoT, Ministry of Education, School of Automation, Guangdong University of Technology（低空感知与探测技术及智能物联网工程研究中心，教育部，自动化学院，广东工业大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）
Zhejiang University（浙江大学）；University of Delaware（特拉华大学）
This paper presents a tightly-coupled multi-sensor fusion algorithm termed LiDAR-inertial-camera fusion (LIC-Fusion), which efficiently fuses IMU measurements, sparse visual features, and extracted LiDAR points. In particular, the proposed LIC-Fusion performs online spatial and temporal sensor calibration between all three asynchronous sensors, in order to compensate for possible calibration variations. The key contribution is the optimal (up to linearization errors) multi-modal sensor fusion of detected and tracked sparse edge/surf feature points from LiDAR scans within an efficient MSCKF-bas…
This paper presents an online initialization method for bootstrapping the optimization-based monocular visual-inertial odometry (VIO). The method can online calibrate the relative transformation (spatial) and time offsets (temporal) among camera and IMU, as well as estimate the initial values of metric scale, velocity, gravity, gyroscope bias, and accelerometer bias during the initialization stage. To compensate for the impact of time offset, our method includes two short-term motion interpolation algorithms for the camera and IMU pose estimation. Besides, it includes a three-step process to i…
FSGlove: An Inertial-Based Hand Tracking System with Shape-Aware Calibration
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（电子信息与电气工程学院，上海交通大学）；Meta Robotics Institute, Shanghai Jiao Tong University（元机器人研究所，上海交通大学）；Qing Yuan Research Institute（清元研究院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
MAC-I$^2$: Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；University of Toronto Institute for Aerospace Studies (UTIAS), University of Toronto（多伦多大学航空航天研究所（多伦多大学））；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
MinJointTracker: Real-time inertial kinematic chain tracking with joint position estimation and minimal state size
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)（莱茵-瓦尔兹技术大学凯泽斯劳滕-兰道）；IT & Engineering, IU Internationale Hochschule Erfurt（IT与工程，国际高等学院埃弗特）
Distributed and Consistent Multi-Robot Visual-Inertial-Ranging Odometry on Lie Groups
School of Control and Computer Engineering, North China Electric Power University（控制与计算机工程学院，华北电力大学）；Department of Electrical and Computer Engineering, George Mason University（电气与计算机工程系，乔治·马歇尔大学）
Joint Magnetometer-IMU Calibration via Maximum A Posteriori Estimation
Dept. of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（电子工程与计算机科学系，皇家理工学院）；Dept. of Electrical Engineering, Linköping University（电子工程系，利尔贝里大学）；Dept. of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, and the Div. of Underwater Technology, Swedish Defence Research Agency (FOI)（电子工程与计算机科学系，皇家理工学院，以及瑞典国防研究机构（FOI）的水下技术系）
We present an unsupervised deep neural network approach to the fusion of RGB-D imagery with inertial measurements for absolute trajectory estimation. Our network, dubbed the Visual-Inertial-Odometry Learner (VIOLearner), learns to perform visual-inertial odometry (VIO) without inertial measurement unit (IMU) intrinsic parameters (corresponding to gyroscope and accelerometer bias or white noise) or the extrinsic calibration between an IMU and camera. The network learns to integrate IMU measurements and generate hypothesis trajectories which are then corrected online according to the Jacobians o…
Hong Kong University of Science and Technology（香港科技大学）
A monocular visual-inertial system (VINS), consisting of a camera and a low-cost inertial measurement unit (IMU), forms the minimum sensor suite for metric six degrees-of-freedom (DOF) state estimation. However, the lack of direct distance measurement poses significant challenges in terms of IMU processing, estimator initialization, extrinsic calibration, and nonlinear optimization. In this work, we present VINS-Mono: a robust and versatile monocular visual-inertial state estimator.Our approach starts with a robust procedure for estimator initialization and failure recovery. A tightly-coupled,…
Large-Scale UWB Anchor Calibration and One-Shot Localization Using Gaussian Process
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）
