When capturing panoramas, people tend to align their cameras with the vertical axis, i.e., the direction of gravity. Moreover, modern devices, such as smartphones and tablets, are equipped with an IMU (Inertial Measurement Unit) that can measure the gravity vector accurately. Using this prior, the y-axes of the cameras can be aligned or assumed to be already aligned, reducing their relative orientation to 1-DOF (degree of freedom). Exploiting this assumption, we propose new minimal solutions to panoramic image stitching of images taken by cameras with coinciding optical centers, i.e., undergoing pure rotation. We consider four practical camera configurations, assuming unknown fixed or varying focal length with or without radial distortion. The solvers are tested both on synthetic scenes and on more than 500k real image pairs from the Sun360 dataset and from scenes captured by us using two smartphones equipped with IMUs. It is shown, that they outperform the state-of-the-art both in terms of accuracy and processing time.
Neural Augmentation Based Panoramic High Dynamic Range Stitching
Wuhan University of Science and Technology（武汉科技大学）；Beihang University（北京航空航天大学）；University of Science and Technology Beijing（北京科技大学）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））
Chungnam National University（忠南大学）；Electronics and Telecommunication Research Institute（电子与电信研究院）
Recently, there has been growing attention on an end-to-end deep learning-based stitching model. However, the most challenging point in deep learning-based stitching is to obtain pairs of input images with a narrow field of view and ground truth images with a wide field of view captured from real-world scenes. To overcome this difficulty, we develop a weakly-supervised learning mechanism to train the stitching model without requiring genuine ground truth images. In addition, we propose a stitching model that takes multiple real-world fisheye images as inputs and creates a 360 output image in a…
Wuhan University of Technology（武汉理工大学）；University of Florida（佛罗里达大学）
Panoramic video is a sort of video recorded at the same point of view to record the full scene. With the development of video surveillance and the requirement for 3D converged video surveillance in smart cities, CPU and GPU are required to possess strong processing abilities to make panoramic video. The traditional panoramic products depend on post processing, which results in high power consumption, low stability and unsatisfying performance in real time. In order to solve these problems,we propose a real-time panoramic video stitching framework.The framework we propose mainly consists of thr…
We study the problem of image alignment for panoramic stitching. Unlike most existing approaches that are feature-based, our algorithm works on pixels directly, and accounts for errors across the whole images globally. Technically, we formulate the alignment problem as rank-1 and sparse matrix decomposition over transformed images, and develop an efficient algorithm for solving this challenging non-convex optimization problem. The algorithm reduces to solving a sequence of subproblems, where we analytically establish exact recovery conditions, convergence and optimality, together with converge…
State Key Laboratory of Information Engineering in Survering, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Xi Rui Optoelectronics Technology Co., Ltd.（西瑞光电科技有限公司）
This paper designs a technique route to generate high-quality panoramic image with depth information, which involves two critical research hotspots: fusion of LiDAR and image data and image stitching. For the fusion of 3D points and image data, since a sparse depth map can be firstly generated by projecting LiDAR point onto the RGB image plane based on our reliable calibrated and synchronized sensors, we adopt a parameter self-adaptive framework to produce 2D dense depth map. For image stitching, optimal seamline for the overlapping area is searched using a graph-cuts-based method to alleviate…
National Yang Ming Chiao Tung University（国立阳明交通大学）；ByteDance（字节跳动）
We propose a novel approach to compute high-resolution (2048x1024 and higher) depths for panoramas that is significantly faster and qualitatively and qualitatively more accurate than the current state-of-the-art method (360MonoDepth). As traditional neural network-based methods have limitations in the output image sizes (up to 1024x512) due to GPU memory constraints, both 360MonoDepth and our method rely on stitching multiple perspective disparity or depth images to come out a unified panoramic depth map. However, to achieve globally consistent stitching, 360MonoDepth relied on solving extensi…
School of Computer Science（计算机科学学院）；The University of Adelaide（阿德莱德大学）
Spatially varying warps are increasingly popular for image alignment. In particular, as-projective-as-possible (APAP) warps have been proven effective for accurate panoramic stitching, especially in cases with significant depth parallax that defeat standard homographic warps. However, estimating spatially varying warps requires a sufficient number of feature matches. In image regions where feature detection or matching fail, the warp loses guidance and is unable to accurately model the true underlying warp, thus resulting in poor registration. In this paper, we propose a correspondence inserti…
The University of Sydney（悉尼大学）；Hong Kong University of Science and Technology（香港科技大学）
In this paper, we propose a panorama stitching algorithm based on asymmetric bidirectional optical flow. This algorithm expects multiple photos captured by fisheye lens cameras as input, and then, through the proposed algorithm, these photos can be merged into a high-quality 360-degree spherical panoramic image. For photos taken from a distant perspective, the parallax among them is relatively small, and the obtained panoramic image can be nearly seamless and undistorted. For photos taken from a close perspective or with a relatively large parallax, a seamless though partially distorted panora…
Institute for Infocomm Research（资讯通信研究院）；University of Glasgow（格拉斯哥大学）；University of Technology and Design（科技设计大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Traditional image stitching algorithms use transforms such as homography to combine different views of a scene. They usually work well when the scene is planar or when the camera is only rotated, keeping its position static. This severely limits their use in real world scenarios where an unmanned aerial vehicle (UAV) potentially hovers around and flies in an enclosed area while rotating to capture a video sequence. We utilize known scene geometry along with recorded camera trajectory to create cylindrical images captured in a given environment such as a tunnel where the camera rotates around i…
It is a high cost problem for panoramic image stitching via image matching algorithm and not practical for real-time performance. In this paper, we take full advantage ofHarris corner invariant characterization method light intensity parallel meaning, translation and rotation, and made a realtime panoramic image stitching algorithm. According to the basic characteristics and performance FPGA classical algorithm, several modules such as the feature point extraction, and matching description is to optimize the feature-based logic. Real-time optimization system to achieve high precision match. Th…
You Need a Transition Plane: Bridging Continuous Panoramic 3D Reconstruction with Perspective Gaussian Splatting
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE（教育部视觉智能+X国际合作联合实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
College of Science and Arts in Qurayyat, Jouf University（焦夫大学古拉亚特文理学院）；Jouf University（焦夫大学）
Most of the current techniques for face recognition require the presence of a full face of the person to be recognized, and this situation is difficult to achieve in practice, the required person may appear with a part of his face, which requires prediction of the part that did not appear. Most of the current forecasting processes are done by what is known as image interpolation, which does not give reliable results, especially if the missing part is large. In this work, we adopted the process of stitching the face by completing the missing part with the flipping of the part shown in the pictu…
In this paper, we derive a new differential homography that can account for the scanline-varying camera poses in Rolling Shutter (RS) cameras, and demonstrate its application to carry out RS-aware image stitching and rectification at one stroke. Despite the high complexity of RS geometry, we focus in this paper on a special yet common input -- two consecutive frames from a video stream, wherein the inter-frame motion is restricted from being arbitrarily large. This allows us to adopt simpler differential motion model, leading to a straightforward and practical minimal solver. To deal with non-…
Center for Combinatorics, Nankai University（南开大学组合数学中心）
Seam-cutting and seam-driven techniques have been proven effective for handling imperfect image series in image stitching. Generally, seam-driven is to utilize seam-cutting to find a best seam from one or finite alignment hypotheses based on a predefined seam quality metric. However, the quality metrics in most methods are defined to measure the average performance of the pixels on the seam without considering the relevance and variance among them. This may cause that the seam with the minimal measure is not optimal (perception-inconsistent) in human perception. In this paper, we propose a nov…
Washington University in St. Louis（圣路易斯华盛顿大学）；Zillow
This paper addresses the problem of Structure from Motion (SfM) for indoor panoramic image streams, extremely challenging even for the state-of-the-art due to the lack of textures and minimal parallax. The key idea is the fusion of single-view and multi-view reconstruction techniques via geometric relationship detection (e.g., detecting 2D lines as coplanar in 3D). Rough geometry suffices to perform such detection, and our approach utilizes rough surface normal estimates from an image-to-normal deep network to discover geometric relationships among lines. The detected relationships provide exa…
Event cameras are a paradigm shift in camera technology. Instead of full frames, the sensor captures a sparse set of events caused by intensity changes. Since only the changes are transferred, those cameras are able to capture quick movements of objects in the scene or of the camera itself. In this work we propose a novel method to perform camera tracking of event cameras in a panoramic setting with three degrees of freedom. We propose a direct camera tracking formulation, similar to state-of-the-art in visual odometry. We show that the minimal information needed for simultaneous tracking and …
School of Artificial Intelligence, Chongqing University of Posts and Telecommunications（重庆邮电大学人工智能学院）；Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University（香港理工大学航空与航空工程系）；School of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）
University of Warwick（华威大学）；University Hospitals Coventry and Warwickshire（考文垂和沃里克大学医院）；The Alan Turing Institute（艾伦·图灵研究所）
Synthetic images can be used for the development and evaluation of deep learning algorithms in the context of limited availability of data. In the field of computational pathology, where histology images are large in size and visual context is crucial, synthesis of large high resolution images via generative modeling is a challenging task. This is due to memory and computational constraints hindering the generation of large images. To address this challenge, we propose a novel SAFRON (Stitching Across the FRONtiers) framework to construct realistic, large high resolution tissue image tiles fro…
The contact specular microscope has a wider angle of view than that of the non-contact specular microscope but still cannot capture an image of the entire cornea. To obtain such an image, it is necessary to prepare film on the parts of the image captured sequentially and combine them to create a complete image. This study proposes a framework to automatically generate an entire corneal image from videos captured using a contact specular microscope. Relatively focused images were extracted from the videos and panoramic compositing was performed. If an entire image can be generated, it is possib…
This work presents a novel approach for robust PCA with total variation regularization for foreground-background separation and denoising on noisy, moving camera video. Our proposed algorithm registers the raw (possibly corrupted) frames of a video and then jointly processes the registered frames to produce a decomposition of the scene into a low-rank background component that captures the static components of the scene, a smooth foreground component that captures the dynamic components of the scene, and a sparse component that can isolate corruptions and other non-idealities. Unlike existing …
