A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The depth estimation model must be tested on a variety of cameras equipped to millions of cars with varying camera geometries. Even within a single car, intrinsics vary due to manufacturing tolerances. Deep learning models are sensitive to these changes, and it is practically infeasible to train and test on each camera variant. As a result, we present novel camera-geometry adaptive multi-scale convolutions which utilize the camera parameters as a conditional input, enabling the model to generalize to previously unseen fisheye cameras. Additionally, we improve the distance estimation by pairwise and patchwise vector-based self-attention encoder networks. We evaluate our approach on the Fisheye WoodScape surround-view dataset, significantly improving over previous approaches. We also show a generalization of our approach across different camera viewing angles and perform extensive experiments to support our contributions. To enable comparison with other approaches, we evaluate the front camera data on the KITTI dataset (pinhole camera images) and achieve state-of-the-art performance among self-supervised monocular methods. An overview video with qualitative results is provided at https://youtu.be/bmX0UcU9wtA. Baseline code and dataset will be made public.
Valeo R&D（法雷奥研发中心）；Valeo Vision Systems（法雷奥视觉系统公司）；Valeo DAR Kronach（法雷奥克罗纳赫DAR）；University of Limerick（利默里克大学）
Object detection is a comprehensively studied problem in autonomous driving. However, it has been relatively less explored in the case of fisheye cameras. The standard bounding box fails in fisheye cameras due to the strong radial distortion, particularly in the image's periphery. We explore better representations like oriented bounding box, ellipse, and generic polygon for object detection in fisheye images in this work. We use the IoU metric to compare these representations using accurate instance segmentation ground truth. We design a novel curved bounding box model that has optimal propert…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；The University of Tokyo（东京大学）；Line Corporation（Line公司）
Estimating the pose of an unmanned aerial vehicle (UAV) or drone is a challenging task. It is useful for many applications such as navigation, surveillance, tracking objects on the ground, and 3D reconstruction. In this work, we present a LiDAR-camera-based relative pose estimation method between a drone and a ground vehicle, using a LiDAR sensor and a fisheye camera on the vehicle's roof and another fisheye camera mounted under the drone. The LiDAR sensor directly observes the drone and measures its position, and the two cameras estimate the relative orientation using indirect observation of …
Valeo DAR Kronach（法雷奥DAR克罗纳赫）；Technische Universität Ilmenau（伊尔默瑙工业大学）；Technische Universität Braunschweig（布伦瑞克工业大学）；Valeo Vision Systems（法雷奥视觉系统）
State-of-the-art self-supervised learning approaches for monocular depth estimation usually suffer from scale ambiguity. They do not generalize well when applied on distance estimation for complex projection models such as in fisheye and omnidirectional cameras. This paper introduces a novel multi-task learning strategy to improve self-supervised monocular distance estimation on fisheye and pinhole camera images. Our contribution to this work is threefold: Firstly, we introduce a novel distance estimation network architecture using a self-attention based encoder coupled with robust semantic fe…
Maynooth University（梅努斯大学）；University of Limerick（利默里克大学）；Valeo Vision Systems（法雷奥视觉系统公司）
Keypoint detection and description is a commonly used building block in computer vision systems particularly for robotics and autonomous driving. However, the majority of techniques to date have focused on standard cameras with little consideration given to fisheye cameras which are commonly used in urban driving and automated parking. In this paper, we propose a novel training and evaluation pipeline for fisheye images. We make use of SuperPoint as our baseline which is a self-supervised keypoint detector and descriptor that has achieved state-of-the-art results on homography estimation. We i…
OPPO US Research Center（OPPO美国研究中心）；InnoPeak Technology, Inc.（InnoPeak技术公司）；Wing LLC（Wing有限责任公司）
In this paper, we deal with the problem of monocular depth estimation for fisheye cameras in a self-supervised manner. A known issue of self-supervised depth estimation is that it suffers in low-light/over-exposure conditions and in large homogeneous regions. To tackle this issue, we propose a novel ordinal distillation loss that distills the ordinal information from a large teacher model. Such a teacher model, since having been trained on a large amount of diverse data, can capture the depth ordering information well, but lacks in preserving accurate scene geometry. Combined with self-supervi…
In this report, we proposed a 3D reconstruction method for the full-view fisheye camera. The camera we used is Ricoh Theta, which captures spherical images and has a wide field of view (FOV). The conventional stereo apporach based on perspective camera model cannot be directly applied and instead we used a spherical camera model to depict the relation between 3D point and its corresponding observation in the image. We implemented a system that can reconstruct the 3D scene using captures from two or more cameras. A GUI is also created to allow users to control the view perspective and obtain a …
Fisheye cameras are commonly employed for obtaining a large field of view in surveillance, augmented reality and in particular automotive applications. In spite of their prevalence, there are few public datasets for detailed evaluation of computer vision algorithms on fisheye images. We release the first extensive fisheye automotive dataset, WoodScape, named after Robert Wood who invented the fisheye camera in 1906. WoodScape comprises of four surround view cameras and nine tasks including segmentation, depth estimation, 3D bounding box detection and soiling detection. Semantic annotation of 4…
OccTrack360: 4D Panoptic Occupancy Tracking from Surround-View Fisheye Cameras
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University（极端光子学与仪器国家重点实验室，浙江大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
Valeo und Schalter und Sensoren GmbH（法雷奥开关与传感器有限公司）；Valeo Vision Systems（法雷奥视觉系统公司）；Technische University, Chemnitz（开姆尼茨工业大学）
Near field depth estimation around a self driving car is an important function that can be achieved by four wide angle fisheye cameras having a field of view of over 180. Depth estimation based on convolutional neural networks (CNNs) produce state of the art results, but progress is hindered because depth annotation cannot be obtained manually. Synthetic datasets are commonly used but they have limitations. For instance, they do not capture the extensive variability in the appearance of objects like vehicles present in real datasets. There is also a domain shift while performing inference on n…
ETH Zürich（苏黎世联邦理工学院）；DSO National Laboratories（DSO国家实验室）；MPI-IS（马克斯·普朗克智能系统研究所）；University of Tübingen（图宾根大学）；Microsoft（微软）；Chalmers University of Technology（查尔姆斯理工大学）
We present a real-time dense geometric mapping algorithm for large-scale environments. Unlike existing methods which use pinhole cameras, our implementation is based on fisheye cameras which have larger field of view and benefit some other tasks including Visual-Inertial Odometry, localization and object detection around vehicles. Our algorithm runs on in-vehicle PCs at 15 Hz approximately, enabling vision-only 3D scene perception for self-driving vehicles. For each synchronized set of images captured by multiple cameras, we first compute a depth map for a reference camera using plane-sweeping…
This work presents a technique for localization of a smart infrastructure node, consisting of a fisheye camera, in a prior map. These cameras can detect objects that are outside the line of sight of the autonomous vehicles (AV) and send that information to AVs using V2X technology. However, in order for this information to be of any use to the AV, the detected objects should be provided in the reference frame of the prior map that the AV uses for its own navigation. Therefore, it is important to know the accurate pose of the infrastructure camera with respect to the prior map. Here we propose …
Valeo DAR Kronach（法雷奥达尔斯克罗纳赫）；ENSTA ParisTech Palaiseau（巴黎高等理工学院帕莱索校区）；Valeo Vision Systems（法雷奥视觉系统公司）；Technische Universität Ilmenau（伊尔梅瑙工业大学）
Fisheye cameras are commonly used in applications like autonomous driving and surveillance to provide a large field of view ($>180^{\circ}$). However, they come at the cost of strong non-linear distortions which require more complex algorithms. In this paper, we explore Euclidean distance estimation on fisheye cameras for automotive scenes. Obtaining accurate and dense depth supervision is difficult in practice, but self-supervised learning approaches show promising results and could potentially overcome the problem. We present a novel self-supervised scale-aware framework for learning Euclide…
Dense 3D maps from wide-angle cameras is beneficial to robotics applications such as navigation and autonomous driving. In this work, we propose a real-time dense 3D mapping method for fisheye cameras without explicit rectification and undistortion. We extend the conventional variational stereo method by constraining the correspondence search along the epipolar curve using a trajectory field induced by camera motion. We also propose a fast way of generating the trajectory field without increasing the processing time compared to conventional rectified methods. With our implementation, we were a…
The University of Tokyo（东京大学）；Chiba Institute of Technology（千叶工业大学）；Tohoku University（东北大学）
In this study, we present a method for all-around depth estimation from multiple omnidirectional images for indoor environments. In particular, we focus on plane-sweeping stereo as the method for depth estimation from the images. We propose a new icosahedron-based representation and ConvNets for omnidirectional images, which we name "CrownConv" because the representation resembles a crown made of origami. CrownConv can be applied to both fisheye images and equirectangular images to extract features. Furthermore, we propose icosahedron-based spherical sweeping for generating the cost volume on …
1 Research School of Engineering, The \ National University (ANU), Australia；2 Australia Centre of Excellence for \ Vision (ACRV), Australia
Large field-of-view fisheye lens cameras have attracted more and more researchers' attention in the field of robotics. However, there does not exist a convenient off-the-shelf stereo rectification approach which can be applied directly to fisheye stereo rig. One obvious drawback of existing methods is that the resampling distortion (which is defined as the loss of pixels due to under-sampling and the creation of new pixels due to over-sampling during rectification process) is severe if we want to obtain a rectification with epipolar line (not epipolar circle) constraint. To overcome this weakn…
Lero - the Irish Software Research Centre（Lero——爱尔兰软件研究中心）；Maynooth University（梅努斯大学）；Valeo Vision Systems（法雷奥视觉系统公司）
We present the WoodScape fisheye semantic segmentation challenge for autonomous driving which was held as part of the CVPR 2021 Workshop on Omnidirectional Computer Vision (OmniCV). This challenge is one of the first opportunities for the research community to evaluate the semantic segmentation techniques targeted for fisheye camera perception. Due to strong radial distortion standard models don't generalize well to fisheye images and hence the deformations in the visual appearance of objects and entities needs to be encoded implicitly or as explicit knowledge. This challenge served as a mediu…
Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera
Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）；Simon Fraser University（西蒙菲莎大学）
End-to-end Autonomous Vehicle Following System using Monocular Fisheye Camera
department of Automation, Shanghai Jiao Tong University（自动化系，上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education（系统控制与信息处理重点实验室，教育部）；Global Institute of Future Technology, Shanghai Jiao Tong University（未来技术全球研究院，上海交通大学）
3D Gaussian Splatting with Fisheye Images: Field of View Analysis and Depth-Based Initialization
Department of Computing Sciences, Tampere University（塔尔基大学计算机科学系）；Department of Computer Science, Aalto University（阿尔托大学计算机科学系）；Department of Computer Science and Engineering, University of Oulu（奥卢大学计算机科学与工程系）
Zongmu Technology（纵目科技）；University of Victoria（维多利亚大学）；Xiamen University（厦门大学）
The vehicle re-identification (ReID) plays a critical role in the perception system of autonomous driving, which attracts more and more attention in recent years. However, to our best knowledge, there is no existing complete solution for the surround-view system mounted on the vehicle. In this paper, we argue two main challenges in above scenario: i) In single camera view, it is difficult to recognize the same vehicle from the past image frames due to the fisheye distortion, occlusion, truncation, etc. ii) In multi-camera view, the appearance of the same vehicle varies greatly from different c…
Optical flow estimation is a well-studied topic for automated driving applications. Many outstanding optical flow estimation methods have been proposed, but they become erroneous when tested in challenging scenarios that are commonly encountered. Despite the increasing use of fisheye cameras for near-field sensing in automated driving, there is very limited literature on optical flow estimation with strong lens distortion. Thus we propose and evaluate training strategies to improve a learning-based optical flow algorithm by leveraging the only existing fisheye dataset with optical flow ground …
Wide-angle fisheye cameras are commonly used in automated driving for parking and low-speed navigation tasks. Four of such cameras form a surround-view system that provides a complete and detailed view of the vehicle. These cameras are directly exposed to harsh environmental settings and can get soiled very easily by mud, dust, water, frost. Soiling on the camera lens can severely degrade the visual perception algorithms, and a camera cleaning system triggered by a soiling detection algorithm is increasingly being deployed. While adverse weather conditions, such as rain, are getting attention …
Reliable knowledge of road boundaries is critical for autonomous vehicle navigation. We propose a robust curb detection and filtering technique based on the fusion of camera semantics and dense lidar point clouds. The lidar point clouds are collected by fusing multiple lidars for robust feature detection. The camera semantics are based on a modified EfficientNet architecture which is trained with labeled data collected from onboard fisheye cameras. The point clouds are associated with the closest curb segment with $L_2$-norm analysis after projecting into the image space with the fisheye model…
Self-supervised monocular depth estimation has been widely investigated to estimate depth images and relative poses from RGB images. This framework is attractive for researchers because the depth and pose networks can be trained from just time sequence images without the need for the ground truth depth and poses. In this work, we estimate the depth around a robot (360 degree view) using time sequence spherical camera images, from a camera whose parameters are unknown. We propose a learnable axisymmetric camera model which accepts distorted spherical camera images with two fisheye camera images…
Valeo Vision Systems（法雷奥视觉系统公司）；University of Limerick（利默里克大学）
We introduce a visual motion segmentation method employing spherical geometry for fisheye cameras and automoated driving. Three commonly used geometric constraints in pin-hole imagery (the positive height, positive depth and epipolar constraints) are reformulated to spherical coordinates, making them invariant to specific camera configurations as long as the camera calibration is known. A fourth constraint, known as the anti-parallel constraint, is added to resolve motion-parallax ambiguity, to support the detection of moving objects undergoing parallel or near-parallel motion with respect to …
Semi-SMD: Semi-Supervised Metric Depth Estimation via Surrounding Cameras for Autonomous Driving
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou)（机器人与自主系统方向，香港科学与技术大学（广州））；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）
SDGE: Stereo Guided Depth Estimation for 360$^\circ$ Camera Sets
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；SZ DJI Technology Co., Ltd.（深圳市大疆创新科技有限公司）；The University of New South Wales（新南威尔士大学）
This paper proposes a robust localization system that employs deep learning for better scene representation, and enhances the accuracy of 6-DOF camera pose estimation. Inspired by the fact that global scene structure can be revealed by wide field-of-view, we leverage the large overlap of a fisheye camera between adjacent frames, and the powerful high-level feature representations of deep learning. Our main contribution is the novel network architecture that extracts both temporal and spatial information using a Recurrent Neural Network. Specifically, we propose a novel pose regularization term…
Zongmu Technology（纵目科技）；Huazhong University of Science and Technology（华中科技大学）；Technical University of Braunschweig（布伦瑞克工业大学）
Surround-view fisheye perception under valet parking scenes is fundamental and crucial in autonomous driving. Environmental conditions in parking lots perform differently from the common public datasets, such as imperfect light and opacity, which substantially impacts on perception performance. Most existing networks based on public datasets may generalize suboptimal results on these valet parking scenes, also affected by the fisheye distortion. In this article, we introduce a new large-scale fisheye dataset called Fisheye Parking Dataset(FPD) to promote the research in dealing with diverse re…
