Image stitching is typically decomposed into three phases: registration, which aligns the source images with a common target image; seam finding, which determines for each target pixel the source image it should come from; and blending, which smooths transitions over the seams. As described in [1], the seam finding phase attempts to place seams between pixels where the transition between source images is not noticeable. Here, we observe that the most problematic failures of this approach occur when objects are cropped, omitted, or duplicated. We therefore take an object-centered approach to the problem, leveraging recent advances in object detection [2,3,4]. We penalize candidate solutions with this class of error by modifying the energy function used in the seam finding stage. This produces substantially more realistic stitching results on challenging imagery. In addition, these methods can be used to determine when there is non-recoverable occlusion in the input data, and also suggest a simple evaluation metric that can be used to evaluate the output of stitching algorithms.
NavEYE: Vision-Centered Multi-Sensor Fusion-Based Situational Awareness System for Intelligent Surface Vehicles
Sanya Science and Education Innovation Park, Wuhan University of Technology（武汉理工大学三亚科教创新园）；School of Navigation, Wuhan University of Technology（武汉理工大学航运学院）；State Key Laboratory of Maritime Technology and Safety（船舶技术与安全国家重点实验室）
East China University of Science and Technology（华东理工大学）
3D point cloud registration ranks among the most fundamental problems in remote sensing, photogrammetry, robotics and geometric computer vision. Due to the limited accuracy of 3D feature matching techniques, outliers may exist, sometimes even in very large numbers, among the correspondences. Since existing robust solvers may encounter high computational cost or restricted robustness, we propose a novel, fast and highly robust solution, named VOCRA (VOting with Cost function and Rotating Averaging), for the point cloud registration problem with extreme outlier rates. Our first contribution is t…
Beijing Jiaotong University（北京交通大学）；University of Electronic Science and Technology of China（电子科技大学）
Stitched images provide a wide field-of-view (FoV) but suffer from unpleasant irregular boundaries. To deal with this problem, existing image rectangling methods devote to searching an initial mesh and optimizing a target mesh to form the mesh deformation in two stages. Then rectangular images can be generated by warping stitched images. However, these solutions only work for images with rich linear structures, leading to noticeable distortions for portraits and landscapes with non-linear objects. In this paper, we address these issues by proposing the first deep learning solution to image rec…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Washington University in St. Louis（圣路易斯华盛顿大学）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Cornell University（康奈尔大学）
University of Texas – Arlington（德克萨斯大学阿灵顿分校）；Samsung Research America（三星美国研究院）
Dual-fisheye lens cameras have been increasingly used for 360-degree immersive imaging. However, the limited overlapping field of views and misalignment between the two lenses give rise to visible discontinuities in the stitching boundaries. This paper introduces a novel method for dual-fisheye camera stitching that adaptively minimizes the discontinuities in the overlapping regions to generate full spherical 360-degree images. Results show that this approach can produce good quality stitched images for Samsung Gear 360 -- a dual-fisheye camera, even with hard-to-stitch objects in the stitchin…
Depth-Supervised Fusion Network for Seamless-Free Image Stitching
College of Information Science and Technology, Dalian Maritime University（大连海事大学信息科学与技术学院）；School of Software Technology, Dalian University of Technology（大连理工大学软件学院）
Institute for Infocomm Research（资讯通信研究院）；University of Glasgow（格拉斯哥大学）；University of Technology and Design（科技设计大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Traditional image stitching algorithms use transforms such as homography to combine different views of a scene. They usually work well when the scene is planar or when the camera is only rotated, keeping its position static. This severely limits their use in real world scenarios where an unmanned aerial vehicle (UAV) potentially hovers around and flies in an enclosed area while rotating to capture a video sequence. We utilize known scene geometry along with recorded camera trajectory to create cylindrical images captured in a given environment such as a tunnel where the camera rotates around i…
Panorama creation is one of the most widely deployed techniques in computer vision. In addition to industry applications such as Google Street View, it is also used by millions of consumers in smartphones and other cameras. Traditionally, the problem is decomposed into three phases: registration, which picks a single transformation of each source image to align it to the other inputs, seam finding, which selects a source image for each pixel in the final result, and blending, which fixes minor visual artifacts. Here, we observe that the use of a single registration often leads to errors, espec…
University of Maryland, College Park（马里兰大学帕克分校）；Intuitive Surgical, Inc（直观外科公司）
Semantic image segmentation plays an important role in modeling patient-specific anatomy. We propose a convolution neural network, called Kid-Net, along with a training schema to segment kidney vessels: artery, vein and collecting system. Such segmentation is vital during the surgical planning phase in which medical decisions are made before surgical incision. Our main contribution is developing a training schema that handles unbalanced data, reduces false positives and enables high-resolution segmentation with a limited memory budget. These objectives are attained using dynamic weighting, ran…
I-MPN: Inductive Message Passing Network for Efficient Human-in-the-Loop Annotation of Mobile Eye Tracking Data
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Science, VNU-HCM（胡志明市国家大学科学大学）；Quy Nhon AI Research and Development Center, FPT Software（FPT软件归仁人工智能研发中心）；Max Planck Research School for Intelligent Systems (IMPRS-IS)（马克斯·普朗克智能系统研究院）
University of California, Irvine（加州大学尔湾分校）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
The goal of this paper is to compare surface-based and volumetric 3D object shape representations, as well as viewer-centered and object-centered reference frames for single-view 3D shape prediction. We propose a new algorithm for predicting depth maps from multiple viewpoints, with a single depth or RGB image as input. By modifying the network and the way models are evaluated, we can directly compare the merits of voxels vs. surfaces and viewer-centered vs. object-centered for familiar vs. unfamiliar objects, as predicted from RGB or depth images. Among our findings, we show that surface-base…
An Object-Centered Data Acquisition Method for 3D Gaussian Splatting using Mobile Phones
School of Electronics and Information, Northwestern Polytechnical University（电子信息学院，西北工业大学）；College of Engineering, Xi'an International University（工程学院，西安国际大学）
Many telepresence robots are equipped with a forward-facing camera for video communication and a downward-facing camera for navigation. In this paper, we propose to stitch videos from the FF-camera with a wide-angle lens and the DF-camera with a fisheye lens for telepresence robots. We aim at providing more compact and efficient visual feedback for the user interface of telepresence robots with user-friendly interactive experiences. To this end, we present a multi-homography-based video stitching method which stitches videos from a wide-angle camera and a fisheye camera. The method consists of…
Image stitching is challenging in consumer-level photography, due to alignment difficulties in unconstrained shooting environment. Recent studies show that seam-cutting approaches can effectively relieve artifacts generated by local misalignment. Normally, seam-cutting is described in terms of energy minimization, however, few of existing methods consider human perception in their energy functions, which sometimes causes that a seam with minimum energy is not most invisible in the overlapping region. In this paper, we propose a novel perception-based energy function in the seam-cutting framewo…
RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室、计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Baidu（百度）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Ryerson University（瑞尔森大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）
Depth estimation and semantic segmentation play essential roles in scene understanding. The state-of-the-art methods employ multi-task learning to simultaneously learn models for these two tasks at the pixel-wise level. They usually focus on sharing the common features or stitching feature maps from the corresponding branches. However, these methods lack in-depth consideration on the correlation of the geometric cues and the scene parsing. In this paper, we first introduce the concept of semantic objectness to exploit the geometric relationship of these two tasks through an analysis of the ima…
Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；The University of Hong Kong（香港大学）；INSAIT, Sofia University ``St. Kliment Ohridski''（INSAIT，索菲亚大学『圣克莱孟·奥赫里迪斯』）；Shenzhen Loop Area Institute（深圳河套学院）；The Chinese University of Hong Kong（香港中文大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Fudan University（复旦大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Shanghai AI Laboratory（上海人工智能实验室）
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）
Activity detection in surveillance videos is a challenging task caused by small objects, complex activity categories, its untrimmed nature, etc. Existing methods are generally limited in performance due to inaccurate proposals, poor classifiers or inadequate post-processing method. In this work, we propose a comprehensive and effective activity detection system in untrimmed surveillance videos for person-centered and vehicle-centered activities. It consists of four modules, i.e., object localizer, proposal filter, activity classifier and activity refiner. For person-centered activities, a nove…
Human Gaze Boosts Object-Centered Representation Learning
Goethe University Frankfurt（法兰克福大学）；The Hessian Center AI（黑森人工智能中心）；Frankfurt Institute for Advanced Studies（法兰克福高等研究院）；Xidian-FIAS international Joint Research Center（西安电子科技大学-法兰克福高等研究院国际联合研究中心）
We propose a framework stitching of vector representations of large scale raster mosaic images in distributed computing model. In this way, the negative effect of the lack of resources of the central system and scalability problem can be eliminated. The product obtained by this study can be used in applications requiring spatial and temporal analysis on big satellite map images. This study also shows that big data frameworks are not only used in applications of text-based data mining and machine learning algorithms, but also used in applications of algorithms in image processing. The effective…
This paper considers conditions, which allow to preserve important topological and geometric properties in the process of digitization. For this purpose, we introduce a triplet {C,M,D} consisting of a continuous object C, an intermediate model M, which is a collection of subregions whose union is C, a digital model D, which is the intersection graph of M, and apply the consistency principle and criteria of similarity to M in order to make its mathematical structure consistent with the natural structure of D. Specifically, this paper introduces a locally centered lump collection of subregions a…
University of California, Irvine（加州大学欧文分校）；Georgia Institute of Technology（佐治亚理工学院）
We tackle the problem of automatically reconstructing a complete 3D model of a scene from a single RGB image. This challenging task requires inferring the shape of both visible and occluded surfaces. Our approach utilizes viewer-centered, multi-layer representation of scene geometry adapted from recent methods for single object shape completion. To improve the accuracy of view-centered representations for complex scenes, we introduce a novel "Epipolar Feature Transformer" that transfers convolutional network features from an input view to other virtual camera viewpoints, and thus better covers…
University of Science and Technology Beijing（北京科技大学）；Tencent Technology (Shenzhen) Company Limited（腾讯科技（深圳）有限公司）；USTB-EEasyTech Joint Laboratory of Artificial Intelligence（北京科技大学-EEasyTech人工智能联合实验室）；School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Department of Computer Science and Technology, School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院计算机科学与技术系）；Department of Industrial Design, School of Mechanical Engineering, University of Science and Technology Beijing（北京科技大学机械工程学院工业设计系）；Institute of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能研究院）
Text-based visual question answering (VQA) requires to read and understand text in an image to correctly answer a given question. However, most current methods simply add optical character recognition (OCR) tokens extracted from the image into the VQA model without considering contextual information of OCR tokens and mining the relationships between OCR tokens and scene objects. In this paper, we propose a novel text-centered method called RUArt (Reading, Understanding and Answering the Related Text) for text-based VQA. Taking an image and a question as input, RUArt first reads the image and o…
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）；University of Southern California（南加州大学）；Facebook Inc.（脸书公司）
Understanding 3D object structure from a single image is an important but challenging task in computer vision, mostly due to the lack of 3D object annotations to real images. Previous research tackled this problem by either searching for a 3D shape that best explains 2D annotations, or training purely on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Networks (3D-INN), an end-to-end trainable framework that sequentially estimates 2D keypoint heatmaps and 3D object skeletons and poses. Our system learns from both 2D-annotated real images and synthetic 3…
As Super-Resolution (SR) has matured as a research topic, it has been applied to additional topics beyond image reconstruction. In particular, combining classification or object detection tasks with a super-resolution preprocessing stage has yielded improvements in accuracy especially with objects that are small relative to the scene. While SR has shown promise, a study comparing SR and naive upscaling methods such as Nearest Neighbors (NN) interpolation when applied as a preprocessing step for object detection has not been performed. We apply the topic to satellite data and compare the Multi-…
