Conventional absolute camera pose via a Perspective-n-Point (PnP) solver often assumes that the correspondences between 2D image pixels and 3D points are given. When the correspondences between 2D and 3D points are not known a priori, the task becomes the much more challenging blind PnP problem. This paper proposes a deep CNN model which simultaneously solves for both the 6-DoF absolute camera pose and 2D--3D correspondences. Our model comprises three neural modules connected in sequence. First, a two-stream PointNet-inspired network is applied directly to both the 2D image keypoints and the 3D scene points in order to extract discriminative point-wise features harnessing both local and contextual information. Second, a global feature matching module is employed to estimate a matchability matrix among all 2D--3D pairs. Third, the obtained matchability matrix is fed into a classification module to disambiguate inlier matches. The entire network is trained end-to-end, followed by a robust model fitting (P3P-RANSAC) at test time only to recover the 6-DoF camera pose. Extensive tests on both real and simulated data have shown that our method substantially outperforms existing approaches, and is capable of processing thousands of points a second with the state-of-the-art accuracy.
MinCD-PnP: Learning 2D-3D Correspondences with Approximate Blind PnP
Huazhong University of Science and Technology, China（华中科技大学）；Northwestern Polytechnical University, China（西北工业大学）；McMaster University, Canada（麦马士威大学）；Delft University of Technology, Netherlands（代尔夫特理工大学）
Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Blind Perspective-n-Point (PnP) is the problem of estimating the position and orientation of a camera relative to a scene, given 2D image points and 3D scene points, without prior knowledge of the 2D-3D correspondences. Solving for pose and correspondences simultaneously is extremely challenging since the search space is very large. Fortunately it is a coupled problem: the pose can be found easily given the correspondences and vice versa. Existing approaches assume that noisy correspondences are provided, that a good pose prior is available, or that the problem size is small. We instead propos…
University of Maryland（马里兰大学）；Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；IBM Research（IBM研究院）
Blind people have limited access to information about their surroundings, which is important for ensuring one's safety, managing social interactions, and identifying approaching pedestrians. With advances in computer vision, wearable cameras can provide equitable access to such information. However, the always-on nature of these assistive technologies poses privacy concerns for parties that may get recorded. We explore this tension from both perspectives, those of sighted passersby and blind users, taking into account camera visibility, in-person versus remote experience, and extracted visual …
Shanghai Jiao Tong University（上海交通大学）；City University of Hong Kong（香港城市大学）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）
We aim at advancing blind image quality assessment (BIQA), which predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks, in a way that the model parameter sharing and the loss weighting are determined automatically. Specifically, we first describe all candidate label combinations (from multiple tasks) using a textual template, and compute the joint probability from the cosine similarities of the visual-textual embeddings. Predictions of each task can …
Northwestern Polytechnical University（西北工业大学）；Baidu Research（百度研究院）；National Engineering Laboratory of Deep Learning Technology and Application（深度学习技术及应用国家工程实验室）
3D point cloud registration in remote sensing field has been greatly advanced by deep learning based methods, where the rigid transformation is either directly regressed from the two point clouds (correspondences-free approaches) or computed from the learned correspondences (correspondences-based approaches). Existing correspondences-free methods generally learn the holistic representation of the entire point cloud, which is fragile for partial and noisy point clouds. In this paper, we propose a correspondences-free unsupervised point cloud registration (UPCR) method from the representation se…
Institute of Computer Science, Polish Academy of Sciences（波兰科学院计算机科学研究所）
This paper is concerned with possibility of recovery of motion and structure parameters from multiframes under perspective projection when only points on a rigid body are traced. Free (unrestricted and uncontrolled) pattern of motion between frames is assumed. The major question is how many points and/or how many frames are necessary for the task. It has been shown in an earlier paper {Klopotek:95b} that for orthogonal projection two frames are insufficient for the task. The paper demonstrates that, under perspective projection, that total uncertainty about relative position of focal point ver…
Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired b…
This short paper presents a perspective plan to build a null reference image quality assessment. Its main goal is to deliver both the objective score and the distortion map for a given distorted image without the knowledge of its reference image.
Deep learning techniques for blind image super-resolution: A high-scale multi-domain perspective evaluation
Coordenação de Pesquisa Aplicada e Desenvolvimento Tecnológico (COPDT)（应用研究与技术发展协调局（COPDT））；Instituto Nacional de Pesquisas Espaciais (INPE)（国家空间研究所（INPE））
Stereo depth estimation relies on optimal correspondence matching between pixels on epipolar lines in the left and right images to infer depth. In this work, we revisit the problem from a sequence-to-sequence correspondence perspective to replace cost volume construction with dense pixel matching using position information and attention. This approach, named STereo TRansformer (STTR), has several advantages: It 1) relaxes the limitation of a fixed disparity range, 2) identifies occluded regions and provides confidence estimates, and 3) imposes uniqueness constraints during the matching process…
Dalian University of Technology（大连理工大学）；Guilin University of Electronic Technology（桂林电子科技大学）
Blind image deblurring plays a very important role in many vision and multimedia applications. Most existing works tend to introduce complex priors to estimate the sharp image structures for blur kernel estimation. However, it has been verified that directly optimizing these models is challenging and easy to fall into degenerate solutions. Although several experience-based heuristic inference strategies, including trained networks and designed iterations, have been developed, it is still hard to obtain theoretically guaranteed accurate solutions. In this work, a collaborative learning framewor…
AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence
Shanghai Jiao Tong University（上海交通大学）；Tianjin University（天津大学）；Shanghai Second Polytechnic University（上海第二工业大学）
3D inference from monocular vision using neural networks is an important research area of computer vision. Applications of the research area are various with many proposed solutions and have shown remarkable performance. Although many efforts have been invested, there are still unanswered questions, some of which are fundamental. In this paper, I discuss a problem that I hope will come to be known as a generalization of the Blind Perspective-n-Point (Blind PnP) problem for object-driven 3D inference based on 2D representations. The vital difference between the fundamental problem and the Blind…
University of Oulu（奥卢大学）；Czech Technical University in Prague（布拉格捷克理工大学）；ETH Zürich（苏黎世联邦理工学院）
In this paper we study the problem of estimating the semi-generalized pose of a partially calibrated camera, i.e., the pose of a perspective camera with unknown focal length w.r.t. a generalized camera, from a hybrid set of 2D-2D and 2D-3D point correspondences. We study all possible camera configurations within the generalized camera system. To derive practical solvers to previously unsolved challenging configurations, we test different parameterizations as well as different solving strategies based on the state-of-the-art methods for generating efficient polynomial solvers. We evaluate the t…
Institute of Computer Science, Polish Academy of Sciences（波兰科学院计算机科学研究所）
In this paper we would like to deny the results of Wang et al. raising two fundamental claims: * A line does not contribute anything to recognition of motion parameters from two images * Four traceable points are not sufficient to recover motion parameters from two perspective To be constructive, however, we show that four traceable points are sufficient to recover motion parameters from two frames under orthogonal projection and that five points are sufficient to simplify the solution of the two-frame problem under orthogonal projection to solving a linear equation system.
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Google Research（谷歌研究院）
We consider the robust Perspective-n-Point (PnP) problem using a hybrid approach that combines deep learning with model based algorithms. PnP is the problem of estimating the pose of a calibrated camera given a set of 3D points in the world and their corresponding 2D projections in the image. In its more challenging robust version, some of the correspondences may be mismatched and must be efficiently discarded. Classical solutions address PnP via iterative robust non-linear least squares method that exploit the problem's geometry but are either inaccurate or computationally intensive. In contr…
University of Utah（犹他大学）；Instituto Superior Técnico（高等技术学院）
We propose a minimal solution for pose estimation using both points and lines for a multi-perspective camera. In this paper, we treat the multi-perspective camera as a collection of rigidly attached perspective cameras. These type of imaging devices are useful for several computer vision applications that require a large coverage such as surveillance, self-driving cars, and motion-capture studios. While prior methods have considered the cases using solely points or lines, the hybrid case involving both points and lines has not been solved for multi-perspective cameras. We present the solutions…
Huazhong University of Science and Technology（华中科技大学）
This paper presents a thorough evaluation of several widely-used 3D correspondence grouping algorithms, motived by their significance in vision tasks relying on correct feature correspondences. A good correspondence grouping algorithm is desired to retrieve as many as inliers from initial feature matches, giving a rise in both precision and recall. Towards this rule, we deploy the experiments on three benchmarks respectively addressing shape retrieval, 3D object recognition and point cloud registration scenarios. The variety in application context brings a rich category of nuisances including …
Learning Fine-Grained Correspondence with Cross-Perspective Perception for Open-Vocabulary 6D Object Pose Estimation
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心，湖南大学）；State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University（自主智能无人系统国家重点实验室，同济大学）；School of Computer Science and Engineering, Hunan University of Science and Technology（计算机科学与工程学院，湖南科技大学）
Seoul National University（首尔国立大学）；Samsung Electronics Co., Ltd.（三星电子株式会社）
This paper presents a new variational inference framework for image restoration and a convolutional neural network (CNN) structure that can solve the restoration problems described by the proposed framework. Earlier CNN-based image restoration methods primarily focused on network architecture design or training strategy with non-blind scenarios where the degradation models are known or assumed. For a step closer to real-world applications, CNNs are also blindly trained with the whole dataset, including diverse degradations. However, the conditional distribution of a high-quality image given a …
