Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）
Justin Tomasi, Brandon Wagstaff, Steven L. Waslander, Jonathan Kelly
备注：
In IEEE Robotics and Automation Letters (RA-L) and presented at the IEEE International Conference on Robotics and Automation (ICRA'21), Xi'an, China, May 30-Jun. 5, 2021
发表信息：
IEEE Robotics and Automation Letters (RA-L), Vol. 6, No. 2, pp. 2028-2035, Apr. 2021
Successful visual navigation depends upon capturing images that contain sufficient useful information. In this letter, we explore a data-driven approach to account for environmental lighting changes, improving the quality of images for use in visual odometry (VO) or visual simultaneous localization and mapping (SLAM). We train a deep convolutional neural network model to predictively adjust camera gain and exposure time parameters such that consecutive images contain a maximal number of matchable features. The training process is fully self-supervised: our training signal is derived from an underlying VO or SLAM pipeline and, as a result, the model is optimized to perform well with that specific pipeline. We demonstrate through extensive real-world experiments that our network can anticipate and compensate for dramatic lighting changes (e.g., transitions into and out of road tunnels), maintaining a substantially higher number of inlier feature matches than competing camera parameter control algorithms.
Adobe Research（奥多比研究院）；KAIST（韩国科学技术院）；Hyundai Motor Group（现代汽车集团）
We introduce a novel method to automatically adjust camera exposure for image processing and computer vision applications on mobile robot platforms. Because most image processing algorithms rely heavily on low-level image features that are based mainly on local gradient information, we consider that gradient quantity can determine the proper exposure level, allowing a camera to capture the important image features in a manner robust to illumination conditions. We then extend this concept to a multi-camera system and present a new control algorithm to achieve both brightness consistency between…
ETH Zürich（苏黎世联邦理工学院）；Microsoft Artificial Intelligence and Mixed Reality Lab（微软人工智能与混合现实实验室）
Motion blur is one of the major challenges remaining for visual odometry methods. In low-light conditions where longer exposure times are necessary, motion blur can appear even for relatively slow camera motions. In this paper we present a novel hybrid visual odometry pipeline with direct approach that explicitly models and estimates the camera's local trajectory within the exposure time. This allows us to actively compensate for any motion blur that occurs due to the camera motion. In addition, we also contribute a novel benchmarking dataset for motion blur aware visual odometry. In experimen…
Institute of Computing, State University of Campinas - Unicamp（坎皮纳斯州立大学计算研究所）
Classical Visual Simultaneous Localization and Mapping (VSLAM) algorithms can be easily induced to fail when either the robot's motion or the environment is too challenging. The use of Deep Neural Networks to enhance VSLAM algorithms has recently achieved promising results, which we call hybrid methods. In this paper, we compare the performance of hybrid monocular VSLAM methods with different learned feature descriptors. To this end, we propose a set of experiments to evaluate the robustness of the algorithms under different environments, camera motion, and camera sensor noise. Experiments con…
ETH Zurich（苏黎世联邦理工学院）；Voliro Airborne Robotics（Voliro 空中机器人公司）；Sevensense Robotics（Sevensense 机器人公司）
Robust and accurate pose estimation is crucial for many applications in mobile robotics. Extending visual Simultaneous Localization and Mapping (SLAM) with other modalities such as an inertial measurement unit (IMU) can boost robustness and accuracy. However, for a tight sensor fusion, accurate time synchronization of the sensors is often crucial. Changing exposure times, internal sensor filtering, multiple clock sources and unpredictable delays from operation system scheduling and data transfer can make sensor synchronization challenging. In this paper, we present VersaVIS, an Open Versatile …
In this paper, we propose a noise-aware exposure control algorithm for robust robot vision. Our method aims to capture the best-exposed image which can boost the performance of various computer vision and robotics tasks. For this purpose, we carefully design an image quality metric which captures complementary quality attributes and ensures light-weight computation. Specifically, our metric consists of a combination of image gradient, entropy, and noise metrics. The synergy of these measures allows preserving sharp edge and rich texture in the image while maintaining a low noise level. Using t…
Cameras are the most widely exploited sensor in both robotics and computer vision communities. Despite their popularity, two dominant attributes (i.e., gain and exposure time) have been determined empirically and images are captured in very passive manner. In this paper, we present an active and generic camera attribute control scheme using Bayesian optimization. We extend from our previous work [1] in two aspects. First, we propose a method that jointly controls camera gain and exposure time. Secondly, to speed up the Bayesian optimization process, we introduce image synthesis using the camer…
E-3DGS: Gaussian Splatting with Exposure and Motion Events
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University（浙江大学现代光学仪器国家重点实验室）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；The Chair of Robotics, AI, and Real-Time Systems, Technical University of Munich（慕尼黑工业大学机器人、人工智能与实时系统教席）
Hong Kong University of Science and Technology（香港科技大学）；ETH Zurich（苏黎世联邦理工学院）
In this paper, we introduce a method for visual relocalization using the geometric information from a 3D surfel map. A visual database is first built by global indices from the 3D surfel map rendering, which provides associations between image points and 3D surfels. Surfel reprojection constraints are utilized to optimize the keyframe poses and map points in the visual database. A hierarchical camera relocalization algorithm then utilizes the visual database to estimate 6-DoF camera poses. Learned descriptors are further used to improve the performance in challenging cases. We present evaluati…
ECHO: Event-Augmented Context with Hindsight and Outlook for Wrist-Only Manipulation
The Hong Kong University of Science and Technology（香港科技大学）；The University of British Columbia（不列颠哥伦比亚大学）；MMLab, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Astribot；ZENBOT
Hong Kong University of Science and Technology（香港科技大学）
We design a multiscopic vision system that utilizes a low-cost monocular RGB camera to acquire accurate depth estimation. Unlike multi-view stereo with images captured at unconstrained camera poses, the proposed system controls the motion of a camera to capture a sequence of images in horizontally or vertically aligned positions with the same parallax. In this system, we propose a new heuristic method and a robust learning-based method to fuse multiple cost volumes between the reference image and its surrounding images. To obtain training data, we build a synthetic dataset with multiscopic ima…
Modeling cameras for the simulation of autonomous robotics is critical for generating synthetic images with appropriate realism to effectively evaluate a perception algorithm in simulation. In many cases though, simulated images are produced by traditional rendering techniques that exclude or superficially handle processing steps and aspects encountered in the actual camera pipeline. The purpose of this contribution is to quantify the degree to which the exclusion from the camera model of various image generation steps or aspects affect the sim-to-real gap in robotics. We investigate what happ…
In this paper, we propose a multi-objective camera ISP framework that utilizes Deep Reinforcement Learning (DRL) and camera ISP toolbox that consist of network-based and conventional ISP tools. The proposed DRL-based camera ISP framework iteratively selects a proper tool from the toolbox and applies it to the image to maximize a given vision task-specific reward function. For this purpose, we implement total 51 ISP tools that include exposure correction, color-and-tone correction, white balance, sharpening, denoising, and the others. We also propose an efficient DRL network architecture that c…
