In this paper we analyze and compare approaches for 3D reconstruction from low-resolution (250x250), high radial distortion stereo images, which are acquired with small baseline (approximately 1mm). These images are acquired with the system NanEye Stereo manufactured by CMOSIS/AWAIBA. These stereo cameras have also small apertures, which means that high levels of illumination are required. The goal was to develop an approach yielding accurate reconstructions, with a low computational cost, i.e., avoiding non-linear numerical optimization algorithms. In particular we focused on the analysis and comparison of radial distortion models. To perform the analysis and comparison, we defined a baseline method based on available software and methods, such as the Bouguet toolbox [2] or the Computer Vision Toolbox from Matlab. The approaches tested were based on the use of the polynomial model of radial distortion, and on the application of the division model. The issue of the center of distortion was also addressed within the framework of the application of the division model. We concluded that the division model with a single radial distortion parameter has limitations.
1 Research School of Engineering, The \ National University (ANU), Australia；2 Australia Centre of Excellence for \ Vision (ACRV), Australia
Large field-of-view fisheye lens cameras have attracted more and more researchers' attention in the field of robotics. However, there does not exist a convenient off-the-shelf stereo rectification approach which can be applied directly to fisheye stereo rig. One obvious drawback of existing methods is that the resampling distortion (which is defined as the loss of pixels due to under-sampling and the creation of new pixels due to over-sampling during rectification process) is severe if we want to obtain a rectification with epipolar line (not epipolar circle) constraint. To overcome this weakn…
National Tsing Hua University（清华大学）；NEC Labs America（美国电气通信实验室）；National Chiao Tung University（交通大学）
Recently, end-to-end trainable deep neural networks have significantly improved stereo depth estimation for perspective images. However, 360° images captured under equirectangular projection cannot benefit from directly adopting existing methods due to distortion introduced (i.e., lines in 3D are not projected onto lines in 2D). To tackle this issue, we present a novel architecture specifically designed for spherical disparity using the setting of top-bottom 360° camera pairs. Moreover, we propose to mitigate the distortion issue by (1) an additional input branch capturing the position and rel…
Centre for Mathematical Sciences（数学科学中心）；Lund University（隆德大学）
In this paper we present a fast minimal solver for absolute camera pose estimation from four known points that lie in a plane. We assume a perspective camera model with unknown focal length and unknown radial distortion. The radial distortion is modelled using the division model with one parameter. We show that the solutions to this problem can be found from a univariate six-degree polynomial. This results in a very fast and numerically stable solver.
MCPDepth: Omnidirectional Depth Estimation via Stereo Matching from Multi-Cylindrical Panoramas
Washington University in St. Louis（华盛顿大学圣路易斯分校）；The Chinese University of Hong Kong（香港中文大学）；ShanghaiTech University（上海科技大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Multi-view stereo (MVS) reconstruction is essential for creating 3D models. The approach involves applying epipolar rectification followed by dense matching for disparity estimation. However, existing approaches face challenges in applying dense matching for images with different viewpoints primarily due to large differences in object scale. In this paper, we propose a spherical model for epipolar rectification to minimize distortions caused by differences in principal rays. We evaluate the proposed approach using two aerial-based datasets consisting of multi-camera head systems. We show throu…
SDGE: Stereo Guided Depth Estimation for 360$^\circ$ Camera Sets
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；SZ DJI Technology Co., Ltd.（深圳市大疆创新科技有限公司）；The University of New South Wales（新南威尔士大学）
Integrated Vehicle Health Management Centre, Cranfield University（克兰菲尔德大学综合车辆健康管理中心）；School of Aerospace, Cranfield University（克兰菲尔德大学航空学院）
Stereo rectification is the determination of two image transformations (or homographies) that map corresponding points on the two images, projections of the same point in the 3D space, onto the same horizontal line in the transformed images. Rectification is used to simplify the subsequent stereo correspondence problem and speeding up the matching process. Rectifying transformations, in general, introduce perspective distortion on the obtained images, which shall be minimised to improve the accuracy of the following algorithm dealing with the stereo correspondence problem. The search for the o…
A novel algorithm for uncalibrated stereo image-pair rectification under the constraint of geometric distortion, called USR-CGD, is presented in this work. Although it is straightforward to define a rectifying transformation (or homography) given the epipolar geometry, many existing algorithms have unwanted geometric distortions as a side effect. To obtain rectified images with reduced geometric distortions while maintaining a small rectification error, we parameterize the homography by considering the influence of various kinds of geometric distortions. Next, we define several geometric measu…
Although nowadays advanced dense image matching (DIM) algorithms are able to produce LiDAR (Light Detection And Ranging) comparable dense point clouds from satellite stereo images, the accuracy and completeness of such point clouds heavily depend on the geometric parameters of the satellite stereo images. The intersection angle between two images are normally seen as the most important one in stereo data acquisition, as the state-of-the-art DIM algorithms work best on narrow baseline (smaller intersection angle) stereos (E.g. Semi-Global Matching regards 15-25 degrees as good intersection angl…
NICTA（国家信息与通信技术研究院）；Yale-NUS College（耶鲁-新加坡国立大学学院）；The University of Tokyo（东京大学）；NAIST（奈良先端科学技术大学院大学）；Microsoft Research（微软研究院）
This paper introduces depth estimation from water drops. The key idea is that a single water drop adhered to window glass is totally transparent and convex, and thus optically acts like a fisheye lens. If we have more than one water drop in a single image, then through each of them we can see the environment with different view points, similar to stereo. To realize this idea, we need to rectify every water drop imagery to make radially distorted planar surfaces look flat. For this rectification, we consider two physical properties of water drops: (1) A static water drop has constant volume, an…
The Weizmann Institute of Science（魏茨曼科学研究所）；The Open University, Israel（以色列开放大学）
Finding correspondences in wide baseline setups is a challenging problem. Existing approaches have focused largely on developing better feature descriptors for correspondence and on accurate recovery of epipolar line constraints. This paper focuses on the challenging problem of finding correspondences once approximate epipolar constraints are given. We introduce a novel method that integrates a deformation model. Specifically, we formulate the problem as finding the largest number of corresponding points related by a bounded distortion map that obeys the given epipolar constraints. We show tha…
Osaka Prefecture University（大阪府立大学）；Research Institute of Environment, Agriculture and Fisheries, Osaka Prefecture（大阪府环境农林水产综合研究所）
This paper proposes the first object counting method for omnidirectional images. Because conventional object counting methods cannot handle the distortion of omnidirectional images, we propose to process them using stereographic projection, which enables conventional methods to obtain a good approximation of the density function. However, the images obtained by stereographic projection are still distorted. Hence, to manage this distortion, we propose two methods. One is a new data augmentation method designed for the stereographic projection of omnidirectional images. The other is a distortion…
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）；University of Michigan（密歇根大学）
The use of 3D and stereo imaging is rapidly increasing. Compression, transmission, and processing could degrade the quality of stereo images. Quality assessment of such images is different than their 2D counterparts. Metrics that represent 3D perception by human visual system (HVS) are expected to assess stereoscopic quality more accurately. In this paper, inspired by brain sensory/motor fusion process, two stereo images are fused together. Then from every fused image two synthesized images are extracted. Effects of different distortions on statistical distributions of the synthesized images a…
The problem of stereoscopic image quality assessment, which finds applications in 3D visual content delivery such as 3DTV, is investigated in this work. Specifically, we propose a new ParaBoost (parallel-boosting) stereoscopic image quality assessment (PBSIQA) system. The system consists of two stages. In the first stage, various distortions are classified into a few types, and individual quality scorers targeting at a specific distortion type are developed. These scorers offer complementary performance in face of a database consisting of heterogeneous distortion types. In the second stage, sc…
Faculty of Electrical Engineering, Czech Technical University in Prague（捷克理工大学电气工程学院）；ETH Zurich（苏黎世联邦理工学院）；National Institute of Informatics（国立信息学研究所）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克理工大学捷克信息学、机器人与控制论研究所）
The internal geometry of most modern consumer cameras is not adequately described by the perspective projection. Almost all cameras exhibit some radial lens distortion and are equipped with an electronic rolling shutter that induces distortions when the camera moves during the image capture. When focal length has not been calibrated offline, the parameters that describe the radial and rolling shutter distortions are usually unknown. While for global shutter cameras, minimal solvers for the absolute camera pose and unknown focal length and radial distortion are available, solvers for the rollin…
Are Minimal Radial Distortion Solvers Really Necessary for Relative Pose Estimation?
Faculty of Mathematics, Physics and Informatics（数学、物理与信息学系）；Comenius University Bratislava（布拉迪斯拉发科门纽斯大学）；Visual Recognition Group（视觉识别小组）；Faculty of Electrical Engineering（电气工程系）；Czech Technical University in Prague（布拉格捷克技术大学）；Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与自动控制研究所）
Are Minimal Radial Distortion Solvers Necessary for Relative Pose Estimation?
Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague（电气工程学院视觉识别组，捷克技术大学布拉格分校）；Faculty of Mathematics, Physics and Informatics, Comenius University Bratislava（数学、物理和信息学学院，康门纽斯大学布拉迪斯拉发分校）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克信息学、机器人学和自动控制研究所，捷克技术大学布拉格分校）
We propose a novel 3D human pose detector using two panoramic cameras. We show that transforming fisheye perspectives to rectilinear views allows a direct application of two-dimensional deep-learning pose estimation methods, without the explicit need for a costly re-training step to compensate for fisheye image distortions. By utilizing panoramic cameras, our method is capable of accurately estimating human poses over a large field of view. This renders our method suitable for ergonomic analyses and other pose based assessments.
Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Large-scale synthetic datasets are beneficial to stereo matching but usually introduce known domain bias. Although unsupervised image-to-image translation networks represented by CycleGAN show great potential in dealing with domain gap, it is non-trivial to generalize this method to stereo matching due to the problem of pixel distortion and stereo mismatch after translation. In this paper, we propose an end-to-end training framework with domain translation and stereo matching networks to tackle this challenge. First, joint optimization between domain translation and stereo matching networks in…
National Yang Ming Chiao Tung University（国立阳明交通大学）
Preparing appropriate images for camera calibration is crucial to obtain accurate results. In this paper, new suggestions for preparing such data to alleviate the adverse effect of radial distortion for a calibration procedure using principal lines are developed through the investigations of: (i) identifying directions of checkerboard movements in an image which will result in maximum (and minimum) influence on the calibration results, and (ii) inspecting symmetry and monotonicity of such effect in (i) using the above principal lines. Accordingly, it is suggested that the estimation of princip…
Compact and low-cost devices are needed for autonomous driving to image and measure distances to objects 360-degree around. We have been developing an omnidirectional stereo camera exploiting two hyperbolic mirrors and a single set of a lens and sensor, which makes this camera compact and cost efficient. We establish a new calibration method for this camera considering higher-order radial distortion, detailed tangential distortion, an image sensor tilt, and a lens-mirror offset. Our method reduces the calibration error by 6.0 and 4.3 times for the upper- and lower-view images, respectively. Th…
Institute of Space Technology（空间技术研究所）；University of St Andrews（圣安德鲁斯大学）；Univ. Gustave Eiffel（古斯塔夫·埃菲尔大学）；IGN（法国国家地理与林业信息研究所）；ENSG（法国国立地理科学学院）；JIS University（JIS大学）
The Corona KH-4 reconnaissance satellite missions from 1962-1972 acquired panoramic stereo imagery with high spatial resolution of 1.8-7.5 m. The potential of 800,000+ declassified Corona images has not been leveraged due to the complexities arising from handling of panoramic imaging geometry, film distortions and limited availability of the metadata required for georeferencing of the Corona imagery. This paper presents Corona Stereo Pipeline (CoSP): A pipeline for processing of Corona KH-4 stereo panoramic imagery. CoSP utlizes a deep learning based feature matcher SuperGlue to automatically …
In many computer vision domains, the input images must conform with the pinhole camera model, where straight lines in the real world are projected as straight lines in the image. Performing computer vision tasks on live sports broadcast footage imposes challenging requirements where the algorithms cannot rely on a specific calibration pattern must be able to cope with unknown and uncalibrated cameras, radial distortion originating from complex television lenses, few visual clues to compensate distortion by, and the necessity for real-time performance. We present a novel method for single-image…
High-accuracy per-pixel depth is vital for computational photography, so smartphones now have multimodal camera systems with time-of-flight (ToF) depth sensors and multiple color cameras. However, producing accurate high-resolution depth is still challenging due to the low resolution and limited active illumination power of ToF sensors. Fusing RGB stereo and ToF information is a promising direction to overcome these issues, but a key problem remains: to provide high-quality 2D RGB images, the main color sensor's lens is optically stabilized, resulting in an unknown pose for the floating lens t…
Centre for Mathematical Sciences Lund University（隆德大学数学科学中心）；Linköping University（林雪平大学）
In this paper we present a novel algorithm for onboard radial distortion correction for unmanned aerial vehicles (UAVs) equipped with an inertial measurement unit (IMU), that runs in real-time. This approach makes calibration procedures redundant, thus allowing for exchange of optics extemporaneously. By utilizing the IMU data, the cameras can be aligned with the gravity direction. This allows us to work with fewer degrees of freedom, and opens up for further intrinsic calibration. We propose a fast and robust minimal solver for simultaneously estimating the focal length, radial distortion pro…
Maria Curie Sklodowska University（玛丽亚·居里-斯克沃多夫斯卡大学）；Warsaw University of Technology（华沙理工大学）
This paper presents a method for extrinsic camera calibration (estimation of camera rotation and translation matrices) from a sequence of images. It is assumed camera intrinsic matrix and distortion coefficients are known and fixed during the entire sequence. %This allows to decrease a number of pairs of corresponding keypoints between images needed to estimate epipolar geometry compared to uncalibrated case. Performance of the presented method is evaluated on a number of multi-view stereo test datasets. Presented algorithm can be used as a first stage in a dense stereo reconstruction system.
There is a strong demand on capturing underwater scenes without distortions caused by refraction. Since a light field camera can capture several light rays at each point of an image plane from various directions, if geometrically correct rays are chosen, it is possible to synthesize a refraction-free image. In this paper, we propose a novel technique to efficiently select such rays to synthesize a refraction-free image from an underwater image captured by a light field camera. In addition, we propose a stereo technique to reconstruct 3D shapes using a pair of our refraction-free images, which …
University of Chinese Academy of Sciences（中国科学院大学）
Although the recent image-based 3D object detection methods using Pseudo-LiDAR representation have shown great capabilities, a notable gap in efficiency and accuracy still exist compared with LiDAR-based methods. Besides, over-reliance on the stand-alone depth estimator, requiring a large number of pixel-wise annotations in the training stage and more computation in the inferencing stage, limits the scaling application in the real world. In this paper, we propose an efficient and accurate 3D object detection method from stereo images, named RTS3D. Different from the 3D occupancy space in the P…
National University of Singapore（新加坡国立大学）；NEC Laboratories America, Inc.（NEC美国实验室公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning a…
A New Comprehensive Framework for Multi-Exposure Stereo Coding Utilizing Low Rank Tucker-ALS and 3D-HEVC Techniques
Department of Computer Science and Engineering, Thapar Institute of Engineering and Technology（计算机科学与工程系，泰帕尔工程与技术学院）；Department of Computer Science and Information Systems, Work Integrated Learning Programmes, Birla Institute of Technology and Science（计算机科学与信息系统系，比拉理工科学院）
International Prevention Research Institute（国际预防研究所）；Swansea University（斯旺西大学）；Diabetic Eye Screening Wales（威尔士糖尿病眼病筛查中心）；University of Strathclyde Institute of Global Public Health（思克莱德大学全球公共卫生学院）
In order to estimate a registration model of eye fundus images made of an affinity and two radial distortions, we introduce an estimation criterion based on an error between the vessels. In [1], we estimated this model by minimising the error between characteristics points. In this paper, the detected vessels are selected using the circle and ellipse equations of the overlap area boundaries deduced from our model. Our method successfully registers 96 % of the 271 pairs in a Public Health dataset acquired mostly with different cameras. This is better than our previous method [1] and better than…
Tampere University of Technology（坦佩雷理工大学）；Nokia Technologies（诺基亚技术）
We introduced a high-resolution equirectangular panorama (360-degree, virtual reality) dataset for object detection and propose a multi-projection variant of YOLO detector. The main challenge with equirectangular panorama image are i) the lack of annotated training data, ii) high-resolution imagery and iii) severe geometric distortions of objects near the panorama projection poles. In this work, we solve the challenges by i) using training examples available in the "conventional datasets" (ImageNet and COCO), ii) employing only low-resolution images that require only moderate GPU computing pow…
Most of stereo vision works are focusing on computing the dense pixel disparity of a given pair of left and right images. A camera pair usually required lens undistortion and stereo calibration to provide an undistorted epipolar line calibrated image pair for accurate dense pixel disparity computation. Due to noise, object occlusion, repetitive or lack of texture and limitation of matching algorithms, the pixel disparity accuracy usually suffers the most at those object boundary areas. Although statistically the total number of pixel disparity errors might be low (under 2% according to the Kit…
