A Deep Ordinal Distortion Estimation Approach for Distortion Rectification
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network Technology（先进信息科学与网络技术北京市重点实验室）
Distortion is widely existed in the images captured by popular wide-angle cameras and fisheye cameras. Despite the long history of distortion rectification, accurately estimating the distortion parameters from a single distorted image is still challenging. The main reason is these parameters are implicit to image features, influencing the networks to fully learn the distortion information. In this work, we propose a novel distortion rectification approach that can obtain more accurate parameters with higher efficiency. Our key insight is that distortion rectification can be cast as a problem of learning an ordinal distortion from a single distorted image. To solve this problem, we design a local-global associated estimation network that learns the ordinal distortion to approximate the realistic distortion distribution. In contrast to the implicit distortion parameters, the proposed ordinal distortion have more explicit relationship with image features, and thus significantly boosts the distortion perception of neural networks. Considering the redundancy of distortion information, our approach only uses a part of distorted image for the ordinal distortion estimation, showing promising applications in the efficient distortion rectification. To our knowledge, we first unify the heterogeneous distortion parameters into a learning-friendly intermediate representation through ordinal distortion, bridging the gap between image feature and distortion rectification. The experimental results demonstrate that our approach outperforms the state-of-the-art methods by a significant margin, with approximately 23% improvement on the quantitative evaluation while displaying the best performance on visual appearance. The code is available at https://github.com/KangLiao929/OrdinalDistortion.
Nonlinear lens distortion rectification is a common first step in image processing applications where the assumption of a linear camera model is essential. For rectifying the lens distortion, forward distortion model needs to be known. However, many self-calibration methods estimate the inverse distortion model. In the literature, the inverse of the estimated model is approximated for image rectification, which introduces additional error to the system. We propose a novel distortion rectification method that uses the inverse distortion model directly. The method starts by mapping the distorted…
A novel algorithm for uncalibrated stereo image-pair rectification under the constraint of geometric distortion, called USR-CGD, is presented in this work. Although it is straightforward to define a rectifying transformation (or homography) given the epipolar geometry, many existing algorithms have unwanted geometric distortions as a side effect. To obtain rectified images with reduced geometric distortions while maintaining a small rectification error, we parameterize the homography by considering the influence of various kinds of geometric distortions. Next, we define several geometric measu…
This paper presents a novel line-aware rectification network (LaRecNet) to address the problem of fisheye distortion rectification based on the classical observation that straight lines in 3D space should be still straight in image planes. Specifically, the proposed LaRecNet contains three sequential modules to (1) learn the distorted straight lines from fisheye images; (2) estimate the distortion parameters from the learned heatmaps and the image appearance; and (3) rectify the input images via a proposed differentiable rectification layer. To better train and evaluate the proposed model, we …
1 Research School of Engineering, The \ National University (ANU), Australia；2 Australia Centre of Excellence for \ Vision (ACRV), Australia
Large field-of-view fisheye lens cameras have attracted more and more researchers' attention in the field of robotics. However, there does not exist a convenient off-the-shelf stereo rectification approach which can be applied directly to fisheye stereo rig. One obvious drawback of existing methods is that the resampling distortion (which is defined as the loss of pixels due to under-sampling and the creation of new pixels due to over-sampling during rectification process) is severe if we want to obtain a rectification with epipolar line (not epipolar circle) constraint. To overcome this weakn…
Elastic distortion of fingerprints has a negative effect on the performance of fingerprint recognition systems. This negative effect brings inconvenience to users in authentication applications. However, in the negative recognition scenario where users may intentionally distort their fingerprints, this can be a serious problem since distortion will prevent recognition system from identifying malicious users. Current methods aimed at addressing this problem still have limitations. They are often not accurate because they estimate distortion parameters based on the ridge frequency map and orient…
SimFIR: A Simple Framework for Fisheye Image Rectification with Self-supervised Representation Learning
University of Science and Technology of China（中国科学技术大学）；Zhangjiang Laboratory（张江实验室）；The University of Sydney（悉尼大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
提出 SimFIR 自监督框架，利用 ViT 图像块表示与统一畸变感知预训练任务学习细粒度畸变特征，从而提升鱼眼图像校正性能和真实场景泛化能力。
This paper presents a new deep-learning based method to simultaneously calibrate the intrinsic parameters of fisheye lens and rectify the distorted images. Assuming that the distorted lines generated by fisheye projection should be straight after rectification, we propose a novel deep neural network to impose explicit geometry constraints onto processes of the fisheye lens calibration and the distorted image rectification. In addition, considering the nonlinearity of distortion distribution in fisheye images, the proposed network fully exploits multi-scale perception to equalize the rectificat…
Automated recognition of texts in scenes has been a research challenge for years, largely due to the arbitrary variation of text appearances in perspective distortion, text line curvature, text styles and different types of imaging artifacts. The recent deep networks are capable of learning robust representations with respect to imaging artifacts and text style changes, but still face various problems while dealing with scene texts with perspective and curvature distortions. This paper presents an end-to-end trainable scene text recognition system (ESIR) that iteratively removes perspective di…
This paper introduces the first minimal solvers that jointly solve for affine-rectification and radial lens distortion from coplanar repeated patterns. Even with imagery from moderately distorted lenses, plane rectification using the pinhole camera model is inaccurate or invalid. The proposed solvers incorporate lens distortion into the camera model and extend accurate rectification to wide-angle imagery, which is now common from consumer cameras. The solvers are derived from constraints induced by the conjugate translations of an imaged scene plane, which are integrated with the division mode…
Distortion rectification is often required for fisheye images. The generation-based method is one mainstream solution due to its label-free property, but its naive skip-connection and overburdened decoder will cause blur and incomplete correction. First, the skip-connection directly transfers the image features, which may introduce distortion and cause incomplete correction. Second, the decoder is overburdened during simultaneously reconstructing the content and structure of the image, resulting in vague performance. To solve these two problems, in this paper, we focus on the interpretable cor…
University of Sydney（悉尼大学）；National University of Defense Technology（国防科技大学）；Stevens Institute of Technology（斯蒂文斯理工学院）；Hangzhou Dianzi University（杭州电子科技大学）
Images captured by fisheye lenses violate the pinhole camera assumption and suffer from distortions. Rectification of fisheye images is therefore a crucial preprocessing step for many computer vision applications. In this paper, we propose an end-to-end multi-context collaborative deep network for removing distortions from single fisheye images. In contrast to conventional approaches, which focus on extracting hand-crafted features from input images, our method learns high-level semantics and low-level appearance features simultaneously to estimate the distortion parameters. To facilitate trai…
A Deep Single Image Rectification Approach for Pan-Tilt-Zoom Cameras
School of Computer Science, Hubei University of Technology（湖北工业大学计算机学院）；Hubei Key Laboratory of Green Intelligent Computing Power Network（湖北绿色智能算力网络重点实验室）；School of Surveying and Mapping, Wuhan University（武汉大学测绘学院）；Wuhan Tianjihang Information Technology Co., Ltd.（武汉天起航信息技术有限公司）
Numerous significant progress on fisheye image rectification has been achieved through CNN. Nevertheless, constrained by a fixed receptive field, the global distribution and the local symmetry of the distortion have not been fully exploited. To leverage these two characteristics, we introduced Fishformer that processes the fisheye image as a sequence to enhance global and local perception. We tuned the Transformer according to the structural properties of fisheye images. First, the uneven distortion distribution in patches generated by the existing square slicing method confuses the network, r…
Hong Kong UST（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；City University of Hong Kong（香港城市大学）
We propose a novel learning method to rectify document images with various distortion types from a single input image. As opposed to previous learning-based methods, our approach seeks to first learn the distortion flow on input image patches rather than the entire image. We then present a robust technique to stitch the patch results into the rectified document by processing in the gradient domain. Furthermore, we propose a second network to correct the uneven illumination, further improving the readability and OCR accuracy. Due to the less complex distortion present on the smaller image patch…
Cascaded Robust Rectification for Arbitrary Document Images
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家级实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）；The University of Tokyo（东京大学）
The University of Sydney（悉尼大学）；Birkbeck College, University of London（伦敦大学伯贝克学院）
Wide field-of-view (FOV) cameras, which capture a larger scene area than narrow FOV cameras, are used in many applications including 3D reconstruction, autonomous driving, and video surveillance. However, wide-angle images contain distortions that violate the assumptions underlying pinhole camera models, resulting in object distortion, difficulties in estimating scene distance, area, and direction, and preventing the use of off-the-shelf deep models trained on undistorted images for downstream computer vision tasks. Image rectification, which aims to correct these distortions, can solve these …
The University of Sydney（悉尼大学）；UBTECH Sydney AI Centre（优必选悉尼人工智能中心）
Deep learning has demonstrated its power in image rectification by leveraging the representation capacity of deep neural networks via supervised training based on a large-scale synthetic dataset. However, the model may overfit the synthetic images and generalize not well on real-world fisheye images due to the limited universality of a specific distortion model and the lack of explicitly modeling the distortion and rectification process. In this paper, we propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted image…
CMOS sensors employ row-wise acquisition mechanism while imaging a scene, which can result in undesired motion artifacts known as rolling shutter (RS) distortions in the captured image. Existing single image RS rectification methods attempt to account for these distortions by either using algorithms tailored for specific class of scenes which warrants information of intrinsic camera parameters or a learning-based framework with known ground truth motion parameters. In this paper, we propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network…
Czech Institute of Informatics, Robotics and Cybernetics (CIIRC), CTU in Prague（捷克布拉格捷克理工大学信息学、机器人学与控制论研究所（CIIRC））；FEE, CTU in Prague（捷克布拉格捷克理工大学电子工程学院）；ETH Zürich（苏黎世联邦理工学院）
This paper introduces the first minimal solvers that jointly estimate lens distortion and affine rectification from repetitions of rigidly transformed coplanar local features. The proposed solvers incorporate lens distortion into the camera model and extend accurate rectification to wide-angle images that contain nearly any type of coplanar repeated content. We demonstrate a principled approach to generating stable minimal solvers by the Grobner basis method, which is accomplished by sampling feasible monomial bases to maximize numerical stability. Synthetic and real-image experiments confirm …
Scene text images have different shapes and are subjected to various distortions, e.g. perspective distortions. To handle these challenges, the state-of-the-art methods rely on a rectification network, which is connected to the text recognition network. They form a linear pipeline which uses text rectification on all input images, even for images that can be recognized without it. Undoubtedly, the rectification network improves the overall text recognition performance. However, in some cases, the rectification network generates unnecessary distortions on images, resulting in incorrect predicti…
Integrated Vehicle Health Management Centre, Cranfield University（克兰菲尔德大学综合车辆健康管理中心）；School of Aerospace, Cranfield University（克兰菲尔德大学航空学院）
Stereo rectification is the determination of two image transformations (or homographies) that map corresponding points on the two images, projections of the same point in the 3D space, onto the same horizontal line in the transformed images. Rectification is used to simplify the subsequent stereo correspondence problem and speeding up the matching process. Rectifying transformations, in general, introduce perspective distortion on the obtained images, which shall be minimised to improve the accuracy of the following algorithm dealing with the stereo correspondence problem. The search for the o…
University of Science and Technology of China（中国科学技术大学）；The University of Sydney（悉尼大学）
Compared with flatbed scanners, portable smartphones provide more convenience for physical document digitization. However, such digitized documents are often distorted due to uncontrolled physical deformations, camera positions, and illumination variations. To this end, we present DocScanner, a novel framework for document image rectification. Different from existing solutions, DocScanner addresses this issue by introducing a progressive learning mechanism. Specifically, DocScanner maintains a single estimate of the rectified image, which is progressively corrected with a recurrent architectur…
LPTR-AFLNet: Lightweight Integrated Chinese License Plate Rectification and Recognition Network
Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering, China Three Gorges University（湖北水利水电工程智能视觉监测重点实验室，中国三峡大学）；College of Computer and Information Technology, China Three Gorges University（计算机与信息学院，中国三峡大学）；Hubei Key Laboratory of Digital Finance Innovation（湖北省数字金融创新重点实验室）；School of Information Engineering, Hubei University of Economics（信息工程学院，湖北经济学院）
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network（北京先进信息科学与网络重点实验室）
Fisheye image rectification has a long-term unresolved issue with synthetic-to-real generalization. In most previous works, the model trained on the synthetic images obtains unsatisfactory performance on the real-world fisheye image. To this end, we propose a Dual Diffusion Architecture (DDA) for the fisheye rectification with a better generalization ability. The proposed DDA is simultaneously trained with paired synthetic fisheye images and unlabeled real fisheye images. By gradually introducing noises, the synthetic and real fisheye images can eventually develop into a consistent noise distr…
Huazhong University of Science and Technology（华中科技大学）
Recognizing text in natural images is a challenging task with many unsolved problems. Different from those in documents, words in natural images often possess irregular shapes, which are caused by perspective distortion, curved character placement, etc. We propose RARE (Robust text recognizer with Automatic REctification), a recognition model that is robust to irregular text. RARE is a specially-designed deep neural network, which consists of a Spatial Transformer Network (STN) and a Sequence Recognition Network (SRN). In testing, an image is firstly rectified via a predicted Thin-Plate-Spline…
University of Science and Technology of China（中国科学技术大学）；Zhangjiang Laboratory（张江实验室）；The University of Adelaide（阿德莱德大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）
When using images to locate objects, there is the problem of correcting for distortion and misalignment in the images. An elegant way of solving this problem is to generate an error correcting function that maps points in an image to their corrected locations. We generate such a function by fitting a polynomial to a set of sample points. The objective is to identify a polynomial that passes "sufficiently close" to these points with "good" approximation of intermediate points. In the past, it has been difficult to achieve good global polynomial approximation using only sample points. We report …
Unfolder: Fast localization and image rectification of a document with a crease from folding in half
Smart Engines Service LLC（Smart Engines服务有限责任公司）；Institute for Information Transmission Problems of RAS (Kharkevich Institute)（俄罗斯科学院信息传输问题研究所（哈尔凯维奇研究所））；Federal Research Center "Computer Science and Control" of the Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）
Harbin Institute of Technology, Weihai（哈尔滨工业大学（威海））
Geometric rectification of images of distorted documents finds wide applications in document digitization and Optical Character Recognition (OCR). Although smoothly curved deformations have been widely investigated by many works, the most challenging distortions, e.g. complex creases and large foldings, have not been studied in particular. The performance of existing approaches, when applied to largely creased or folded documents, is far from satisfying, leaving substantial room for improvement. To tackle this task, knowledge about document rectification should be incorporated into the computa…
In many computer vision domains, the input images must conform with the pinhole camera model, where straight lines in the real world are projected as straight lines in the image. Performing computer vision tasks on live sports broadcast footage imposes challenging requirements where the algorithms cannot rely on a specific calibration pattern must be able to cope with unknown and uncalibrated cameras, radial distortion originating from complex television lenses, few visual clues to compensate distortion by, and the necessity for real-time performance. We present a novel method for single-image…
Facebook Reality Labs（Facebook现实实验室）；ETH Zürich（苏黎世联邦理工学院）；Ukrainian Catholic University（乌克兰天主教大学）；Czech Technical University in Prague（布拉格捷克理工大学）
This paper introduces the first minimal solvers that jointly estimate lens distortion and affine rectification from the image of rigidly-transformed coplanar features. The solvers work on scenes without straight lines and, in general, relax strong assumptions about scene content made by the state of the art. The proposed solvers use the affine invariant that coplanar repeats have the same scale in rectified space. The solvers are separated into two groups that differ by how the equal scale invariant of rectified space is used to place constraints on the lens undistortion and rectification para…
The University of Tokyo（东京大学）；RIISE；National Institute of Informatics（国立信息学研究所）；University of Maryland College Park（马里兰大学帕克分校）；Hefei Normal University（合肥师范学院）
Most computer vision systems assume distortion-free images as inputs. The widely used rolling-shutter (RS) image sensors, however, suffer from geometric distortion when the camera and object undergo motion during capture. Extensive researches have been conducted on correcting RS distortions. However, most of the existing work relies heavily on the prior assumptions of scenes or motions. Besides, the motion estimation steps are either oversimplified or computationally inefficient due to the heavy flow warping, limiting their applicability. In this paper, we investigate using rolling shutter wit…
UNC at Chapel Hill（北卡罗来纳大学教堂山分校）；Ricoh Innovations（理光创新公司）；SAIC Innovation Center（上汽集团创新中心）；Stanford University（斯坦福大学）
Recently, convolutional neural networks (CNN) have been successfully applied to view synthesis problems. However, such CNN-based methods can suffer from lack of texture details, shape distortions, or high computational complexity. In this paper, we propose a novel CNN architecture for view synthesis called "Deep View Morphing" that does not suffer from these issues. To synthesize a middle view of two input images, a rectification network first rectifies the two input images. An encoder-decoder network then generates dense correspondences between the rectified images and blending masks to predi…
Shanghai Jiao Tong University（上海交通大学）；Texas A&M University（得克萨斯农工大学）；University of Macau（澳门大学）
Liquify is a common technique for image editing, which can be used for image distortion. Due to the uncertainty in the distortion variation, restoring distorted images caused by liquify filter is a challenging task. To edit images in an efficient way, distorted images are expected to be restored automatically. This paper aims at the distorted image restoration, which is characterized by seeking the appropriate warping and completion of a distorted image. Existing methods focus on the hardware assistance or the geometric principle to solve the specific regular deformation caused by natural phen…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence of Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
As camera-based documents are increasingly used, the rectification of distorted document images becomes a need to improve the recognition performance. In this paper, we propose a novel framework for both rectifying distorted document image and removing background finely, by estimating pixel-wise displacements using a fully convolutional network (FCN). The document image is rectified by transformation according to the displacements of pixels. The FCN is trained by regressing displacements of synthesized distorted documents, and to control the smoothness of displacements, we propose a Local Smoo…
In subjective full-reference image quality assessment, differences between perceptual image qualities of the reference image and its distorted versions are evaluated, often using degradation category ratings (DCR). However, the DCR has been criticized since differences between rating categories on this ordinal scale might not be perceptually equidistant, and observers may have different understandings of the categories. Pair comparisons (PC) of distorted images, followed by Thurstonian reconstruction of scale values, overcome these problems. In addition, PC is more sensitive than DCR, and it c…
