Compact and low-cost devices are needed for autonomous driving to image and measure distances to objects 360-degree around. We have been developing an omnidirectional stereo camera exploiting two hyperbolic mirrors and a single set of a lens and sensor, which makes this camera compact and cost efficient. We establish a new calibration method for this camera considering higher-order radial distortion, detailed tangential distortion, an image sensor tilt, and a lens-mirror offset. Our method reduces the calibration error by 6.0 and 4.3 times for the upper- and lower-view images, respectively. The random error of the distance measurement is 4.9% and the systematic error is 5.7% up to objects 14 meters apart, which is improved almost nine times compared to the conventional method. The remaining distance errors is due to a degraded optical resolution of the prototype, which we plan to make further improvements as future work.
EPFL（洛桑联邦理工学院）；Visual Intelligence for Transportation (VITA) lab（交通视觉智能实验室）
Monocular and stereo visions are cost-effective solutions for 3D human localization in the context of self-driving cars or social robots. However, they are usually developed independently and have their respective strengths and limitations. We propose a novel unified learning framework that leverages the strengths of both monocular and stereo cues for 3D human localization. Our method jointly (i) associates humans in left-right images, (ii) deals with occluded and distant cases in stereo settings by relying on the robustness of monocular cues, and (iii) tackles the intrinsic ambiguity of monoc…
Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Pseudo-LiDAR 3D detectors have made remarkable progress in monocular 3D detection by enhancing the capability of perceiving depth with depth estimation networks, and using LiDAR-based 3D detection architectures. The advanced stereo 3D detectors can also accurately localize 3D objects. The gap in image-to-image generation for stereo views is much smaller than that in image-to-LiDAR generation. Motivated by this, we propose a Pseudo-Stereo 3D detection framework with three novel virtual view generation methods, including image-level generation, feature-level generation, and feature-clone, for de…
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Jiaotong University（北京交通大学）
Estimating the depth of omnidirectional images is more challenging than that of normal field-of-view (NFoV) images because the varying distortion can significantly twist an object's shape. The existing methods suffer from troublesome distortion while estimating the depth of omnidirectional images, leading to inferior performance. To reduce the negative impact of the distortion influence, we propose a distortion-tolerant omnidirectional depth estimation algorithm using a dual-cubemap. It comprises two modules: Dual-Cubemap Depth Estimation (DCDE) module and Boundary Revision (BR) module. In DCD…
Depth estimation is essential for various important real-world applications such as autonomous driving. However, it suffers from severe performance degradation in high-velocity scenario since traditional cameras can only capture blurred images. To deal with this problem, the spike camera is designed to capture the pixel-wise luminance intensity at high frame rate. However, depth estimation with spike camera remains very challenging using traditional monocular or stereo depth estimation algorithms, which are based on the photometric consistency. In this paper, we propose a novel Uncertainty-Gui…
The Hong Kong University of Science and Technology（香港科技大学）
We propose a 3D object detection system with multi-sensor refinement in the context of autonomous driving. In our framework, the monocular camera serves as the fundamental sensor for 2D object proposal and initial 3D bounding box prediction. While the stereo cameras and LiDAR are treated as adaptive plug-in sensors to refine the 3D box localization performance. For each observed element in the raw measurement domain (e.g., pixels for stereo, 3D points for LiDAR), we model the local geometry as an instance vector representation, which indicates the 3D coordinate of each element respecting to th…
School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）；University of Western Australia（西澳大利亚大学）；Alibaba Group（阿里巴巴集团）
A main challenge for tasks on panorama lies in the distortion of objects among images. In this work, we propose a Distortion-Aware Monocular Omnidirectional (DAMO) dense depth estimation network to address this challenge on indoor panoramas with two steps. First, we introduce a distortion-aware module to extract calibrated semantic features from omnidirectional images. Specifically, we exploit deformable convolution to adjust its sampling grids to geometric variations of distorted objects on panoramas and then utilize a strip pooling module to sample against horizontal distortion introduced by…
Perceptual Depth Quality Assessment of Stereoscopic Omnidirectional Images
School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；Department of Electrical and Computer Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）
Although the number of camera-based sensors mounted on vehicles has recently increased dramatically, robust and accurate object velocity detection is difficult. Additionally, it is still common to use radar as a fusion system. We have developed a method to accurately detect the velocity of object using a camera, based on a large-scale dataset collected over 20 years by the automotive manufacturer, SUBARU. The proposed method consists of three methods: an High Dynamic Range (HDR) detection method that fuses multiple stereo disparity images, a fusion method that combines the results of monocular…
Understanding the world in 3D is a critical component of urban autonomous driving. Generally, the combination of expensive LiDAR sensors and stereo RGB imaging has been paramount for successful 3D object detection algorithms, whereas monocular image-only methods experience drastically reduced performance. We propose to reduce the gap by reformulating the monocular 3D detection problem as a standalone 3D region proposal network. We leverage the geometric relationship of 2D and 3D perspectives, allowing 3D boxes to utilize well-known and powerful convolutional features generated in the image-spa…
Monocular scene reconstruction is essential for modern applications such as robotics or autonomous driving. Although stereo methods usually result in better accuracy than monocular methods, they are more expensive and more difficult to calibrate. In this work, we present a novel second order optimal minimum energy filter that jointly estimates the camera motion, the disparity map and also higher order kinematics recursively on a product Lie group containing a novel disparity group. This mathematical framework enables to cope with non-Euclidean state spaces, non-linear observations and high dim…
Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention
School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology（全州科学技术院电气工程与计算机科学学院）；Division of National Science and Technology Data (Large Scale AI Research Group), Korea Institute of Science and Technology Information (KISTI)（国家科学技术数据 division（大规模人工智能研究组），韩国科学技术信息研究院）；Department of Information Systems, College of Computer Sciences and Information Technology, King Faisal University（国王法尔萨大学信息系统系，计算机科学与信息科技学院）
Accurate depth estimation with lowest compute and energy cost is a crucial requirement for unmanned and battery operated autonomous systems. Robotic applications require real time depth estimation for navigation and decision making under rapidly changing 3D surroundings. A high accuracy algorithm may provide the best depth estimation but may consume tremendous compute and energy resources. A general trade-off is to choose less accurate methods for initial depth estimate and a more accurate yet compute intensive method when needed. Previous work has shown this trade-off can be improved by devel…
Objective quality assessment of stereoscopic panoramic images becomes a challenging problem owing to the rapid growth of 360-degree contents. Different from traditional 2D image quality assessment (IQA), more complex aspects are involved in 3D omnidirectional IQA, especially unlimited field of view (FoV) and extra depth perception, which brings difficulty to evaluate the quality of experience (QoE) of 3D omnidirectional images. In this paper, we propose a multi-viewport based fullreference stereo 360 IQA model. Due to the freely changeable viewports when browsing in the head-mounted display (H…
University of Adelaide（阿德莱德大学）；Queensland University of Technology（昆士兰科技大学）
Intra-operative automatic semantic segmentation of knee joint structures can assist surgeons during knee arthroscopy in terms of situational awareness. However, due to poor imaging conditions (e.g., low texture, overexposure, etc.), automatic semantic segmentation is a challenging scenario, which justifies the scarce literature on this topic. In this paper, we propose a novel self-supervised monocular depth estimation to regularise the training of the semantic segmentation in knee arthroscopy. To further regularise the depth estimation, we propose the use of clean training images captured by t…
