Wide field-of-view (FOV) cameras, which capture a larger scene area than narrow FOV cameras, are used in many applications including 3D reconstruction, autonomous driving, and video surveillance. However, wide-angle images contain distortions that violate the assumptions underlying pinhole camera models, resulting in object distortion, difficulties in estimating scene distance, area, and direction, and preventing the use of off-the-shelf deep models trained on undistorted images for downstream computer vision tasks. Image rectification, which aims to correct these distortions, can solve these problems. In this paper, we comprehensively survey progress in wide-angle image rectification from transformation models to rectification methods. Specifically, we first present a detailed description and discussion of the camera models used in different approaches. Then, we summarize several distortion models including radial distortion and projection distortion. Next, we review both traditional geometry-based image rectification methods and deep learning-based methods, where the former formulate distortion parameter estimation as an optimization problem and the latter treat it as a regression problem by leveraging the power of deep neural networks. We evaluate the performance of state-of-the-art methods on public datasets and show that although both kinds of methods can achieve good results, these methods only work well for specific camera models and distortion types. We also provide a strong baseline model and carry out an empirical study of different distortion models on synthetic datasets and real-world wide-angle images. Finally, we discuss several potential research directions that are expected to further advance this area in the future.
Dense 3D maps from wide-angle cameras is beneficial to robotics applications such as navigation and autonomous driving. In this work, we propose a real-time dense 3D mapping method for fisheye cameras without explicit rectification and undistortion. We extend the conventional variational stereo method by constraining the correspondence search along the epipolar curve using a trajectory field induced by camera motion. We also propose a fast way of generating the trajectory field without increasing the processing time compared to conventional rectified methods. With our implementation, we were a…
The ubiquity of smartphone cameras has led to more and more documents being captured by cameras rather than scanned. Unlike flatbed scanners, photographed documents are often folded and crumpled, resulting in large local variance in text structure. The problem of document rectification is fundamental to the Optical Character Recognition (OCR) process on documents, and its ability to overcome geometric distortions significantly affects recognition accuracy. Despite the great progress in recent OCR systems, most still rely on a pre-process that ensures the text lines are straight and axis aligne…
A Deep Single Image Rectification Approach for Pan-Tilt-Zoom Cameras
School of Computer Science, Hubei University of Technology（湖北工业大学计算机学院）；Hubei Key Laboratory of Green Intelligent Computing Power Network（湖北绿色智能算力网络重点实验室）；School of Surveying and Mapping, Wuhan University（武汉大学测绘学院）；Wuhan Tianjihang Information Technology Co., Ltd.（武汉天起航信息技术有限公司）
Czech Institute of Informatics, Robotics and Cybernetics (CIIRC), CTU in Prague（捷克布拉格捷克理工大学信息学、机器人学与控制论研究所（CIIRC））；FEE, CTU in Prague（捷克布拉格捷克理工大学电子工程学院）；ETH Zürich（苏黎世联邦理工学院）
This paper introduces the first minimal solvers that jointly estimate lens distortion and affine rectification from repetitions of rigidly transformed coplanar local features. The proposed solvers incorporate lens distortion into the camera model and extend accurate rectification to wide-angle images that contain nearly any type of coplanar repeated content. We demonstrate a principled approach to generating stable minimal solvers by the Grobner basis method, which is accomplished by sampling feasible monomial bases to maximize numerical stability. Synthetic and real-image experiments confirm …
This paper introduces the first minimal solvers that jointly solve for affine-rectification and radial lens distortion from coplanar repeated patterns. Even with imagery from moderately distorted lenses, plane rectification using the pinhole camera model is inaccurate or invalid. The proposed solvers incorporate lens distortion into the camera model and extend accurate rectification to wide-angle imagery, which is now common from consumer cameras. The solvers are derived from constraints induced by the conjugate translations of an imaged scene plane, which are integrated with the division mode…
Carnegie Mellon University（卡内基梅隆大学）；Volvo Construction Equipment（沃尔沃建筑设备）
Across a majority of pedestrian detection datasets, it is typically assumed that pedestrians will be standing upright with respect to the image coordinate system. This assumption, however, is not always valid for many vision-equipped mobile platforms such as mobile phones, UAVs or construction vehicles on rugged terrain. In these situations, the motion of the camera can cause images of pedestrians to be captured at extreme angles. This can lead to very poor pedestrian detection performance when using standard pedestrian detectors. To address this issue, we propose a Rotational Rectification Ne…
Harbin Institute of Technology, Weihai（哈尔滨工业大学（威海））
Geometric rectification of images of distorted documents finds wide applications in document digitization and Optical Character Recognition (OCR). Although smoothly curved deformations have been widely investigated by many works, the most challenging distortions, e.g. complex creases and large foldings, have not been studied in particular. The performance of existing approaches, when applied to largely creased or folded documents, is far from satisfying, leaving substantial room for improvement. To tackle this task, knowledge about document rectification should be incorporated into the computa…
A Deep Ordinal Distortion Estimation Approach for Distortion Rectification
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network Technology（先进信息科学与网络技术北京市重点实验室）
Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院“计算机科学与控制”联邦研究中心）；Moscow Institute of Physics and Technology (State University)（莫斯科物理技术学院（国立大学））；The Institute for Information Transmission Problems of Russian Academy of Sciences（俄罗斯科学院信息传输问题研究所）；Smart Engines Service LLC（Smart Engines服务有限责任公司）
This paper presents a method for metric rectification of planar objects that preserves angles and length ratios. An inner structure of an object is assumed to follow the laws of Manhattan World i.e. the majority of line segments are aligned with two orthogonal directions of the object. For that purpose we introduce the method that estimates the position of two vanishing points corresponding to the main object directions. It is based on an original optimization function of segments that estimates a vanishing point position. For calculation of the rectification homography with two vanishing poin…
National ICT Australia (NICTA)（澳大利亚国家信息与通信技术研究院（NICTA））；Australian National University（澳大利亚国立大学）；Osaka University（大阪大学）；Microsoft Research Redmond（微软雷德蒙德研究院）；Microsoft, Japan（微软（日本））；Microsoft Research Asia（微软亚洲研究院）
Digitally unwrapping images of paper sheets is crucial for accurate document scanning and text recognition. This paper presents a method for automatically rectifying curved or folded paper sheets from a few images captured from multiple viewpoints. Prior methods either need expensive 3D scanners or model deformable surfaces using over-simplified parametric representations. In contrast, our method uses regular images and is based on general developable surface models that can represent a wide variety of paper deformations. Our main contribution is a new robust rectification method based on ridg…
LPTR-AFLNet: Lightweight Integrated Chinese License Plate Rectification and Recognition Network
Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering, China Three Gorges University（湖北水利水电工程智能视觉监测重点实验室，中国三峡大学）；College of Computer and Information Technology, China Three Gorges University（计算机与信息学院，中国三峡大学）；Hubei Key Laboratory of Digital Finance Innovation（湖北省数字金融创新重点实验室）；School of Information Engineering, Hubei University of Economics（信息工程学院，湖北经济学院）
Department of Electrical and Computer Engineering, Northwestern University（电气与计算机工程系，西北大学）；Department of Materials Science and Engineering, Northwestern University（材料科学与工程系，西北大学）；Applied Physics Program, Northwestern University（应用物理项目，西北大学）
Facebook Reality Labs（Facebook现实实验室）；ETH Zürich（苏黎世联邦理工学院）；Ukrainian Catholic University（乌克兰天主教大学）；Czech Technical University in Prague（布拉格捷克理工大学）
This paper introduces the first minimal solvers that jointly estimate lens distortion and affine rectification from the image of rigidly-transformed coplanar features. The solvers work on scenes without straight lines and, in general, relax strong assumptions about scene content made by the state of the art. The proposed solvers use the affine invariant that coplanar repeats have the same scale in rectified space. The solvers are separated into two groups that differ by how the equal scale invariant of rectified space is used to place constraints on the lens undistortion and rectification para…
Czech Technical University in Prague（布拉格捷克理工大学）；ETH Zurich（苏黎世联邦理工学院）；Ukrainian Catholic University（乌克兰天主教大学）
This paper introduces minimal solvers that jointly solve for radial lens undistortion and affine-rectification using local features extracted from the image of coplanar translated and reflected scene texture, which is common in man-made environments. The proposed solvers accommodate different types of local features and sampling strategies, and three of the proposed variants require just one feature correspondence. State-of-the-art techniques from algebraic geometry are used to simplify the formulation of the solvers. The generated solvers are stable, small and fast. Synthetic and real-image e…
Wide-angle cameras are uniquely positioned for mobile robots, by virtue of the rich information they provide in a small, light, and cost-effective form factor. An accurate calibration of the intrinsics and extrinsics is a critical pre-requisite for using the edge of a wide-angle lens for depth perception and odometry. Calibrating wide-angle lenses with current state-of-the-art techniques yields poor results due to extreme distortion at the edge, as most algorithms assume a lens with low to medium distortion closer to a pinhole projection. In this work we present our methodology for accurate wi…
Video blogs and selfies are popular social media formats, which are often captured by wide-angle cameras to show human subjects and expanded background. Unfortunately, due to perspective projection, faces near corners and edges exhibit apparent distortions that stretch and squish the facial features, resulting in poor video quality. In this work, we present a video warping algorithm to correct these distortions. Our key idea is to apply stereographic projection locally on the facial regions. We formulate a mesh warp problem using spatial-temporal energy minimization and minimize background def…
