Keypoint detection and description is a commonly used building block in computer vision systems particularly for robotics and autonomous driving. However, the majority of techniques to date have focused on standard cameras with little consideration given to fisheye cameras which are commonly used in urban driving and automated parking. In this paper, we propose a novel training and evaluation pipeline for fisheye images. We make use of SuperPoint as our baseline which is a self-supervised keypoint detector and descriptor that has achieved state-of-the-art results on homography estimation. We introduce a fisheye adaptation pipeline to enable training on undistorted fisheye images. We evaluate the performance on the HPatches benchmark, and, by introducing a fisheye based evaluation method for detection repeatability and descriptor matching correctness, on the Oxford RobotCar dataset.
Fast binary descriptors build the core for many vision based applications with real-time demands like object detection, Visual Odometry or SLAM. Commonly it is assumed, that the acquired images and thus the patches extracted around keypoints originate from a perspective projection ignoring image distortion or completely different types of projections such as omnidirectional or fisheye. Usually the deviations from a perfect perspective projection are corrected by undistortion. Latter, however, introduces severe artifacts if the cameras field-of-view gets larger. In this paper, we propose a dist…
Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；University of Jena（耶拿大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；TU Darmstadt（达姆施塔特工业大学）
In this paper we propose a novel 3D single-shot object detection method for detecting vehicles in monocular RGB images. Our approach lifts 2D detections to 3D space by predicting additional regression and classification parameters and hence keeping the runtime close to pure 2D object detection. The additional parameters are transformed to 3D bounding box keypoints within the network under geometric constraints. Our proposed method features a full 3D description including all three angles of rotation without supervision by any labeled ground truth data for the object's orientation, as it focuse…
Toyota Research Institute (TRI)（丰田研究院（TRI））；KTH Royal Institute of Technology（KTH皇家理工学院）
Identifying salient points in images is a crucial component for visual odometry, Structure-from-Motion or SLAM algorithms. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks. However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators. We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair c…
Institute of Product and Process Innovation（产品与流程创新研究所）；Leuphana University Lüneburg（吕讷堡莱奥法纳大学）
Based on different projection geometry, a fisheye image can be presented as a parameterized non-rectilinear image. Deep neural networks(DNN) is one of the solutions to extract parameters for fisheye image feature description. However, a large number of images are required for training a reasonable prediction model for DNN. In this paper, we propose to extend the scale of the training dataset using parameterized synthetic images. It effectively boosts the diversity of images and avoids the data scale limitation. To simulate different viewing angles and distances, we adopt controllable parameter…
JD Finance America Corporation（京东金融美国公司）；University of Missouri（密苏里大学）
Pig counting is a crucial task for large-scale pig farming, which is usually completed by human visually. But this process is very time-consuming and error-prone. Few studies in literature developed automated pig counting method. Existing methods only focused on pig counting using single image, and its accuracy is challenged by several factors, including pig movements, occlusion and overlapping. Especially, the field of view of a single image is very limited, and could not meet the requirements of pig counting for large pig grouping houses. To that end, we presented a real-time automated pig c…
Hanoi University of Science and Technology（河内科技大学）；Viettel High Technology Industries Corporation（Viettel高科技工业公司）；FPT University（FPT大学）
People detection in top-view, fish-eye images is challenging as people in fish-eye images often appear in arbitrary directions and are distorted differently. Due to this unique radial geometry, axis-aligned people detectors often work poorly on fish-eye frames. Recent works account for this variability by modifying existing anchor-based detectors or relying on complex pre/post-processing. Anchor-based methods spread a set of pre-defined bounding boxes on the input image, most of which are invalid. In addition to being inefficient, this approach could lead to a significant imbalance between the…
ETH Zurich（苏黎世联邦理工学院）；Voliro Airborne Robotics（Voliro空中机器人公司）
With the advent of powerful, light-weight 3D LiDARs, they have become the hearth of many navigation and SLAM algorithms on various autonomous systems. Pointcloud registration methods working with unstructured pointclouds such as ICP are often computationally expensive or require a good initial guess. Furthermore, 3D feature-based registration methods have never quite reached the robustness of 2D methods in visual SLAM. With the continuously increasing resolution of LiDAR range images, these 2D methods not only become applicable but should exploit the illumination-independent modalities that co…
In this paper, we explore how three related tasks, namely keypoint detection, description, and image retrieval can be jointly tackled using a single unified framework, which is trained without the need of training data with point to point correspondences. By leveraging diverse information from sequential layers of a standard ResNet-based architecture, we are able to extract keypoints and descriptors that encode local information using generic techniques such as local activation norms, channel grouping and dropping, and self-distillation. Subsequently, global information for image retrieval is …
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shandong University of Science and Technology（山东科技大学）
Keypoint detection and description play a central role in computer vision. Most existing methods are in the form of scene-level prediction, without returning the object classes of different keypoints. In this paper, we propose the object-centric formulation, which, beyond the conventional setting, requires further identifying which object each interest point belongs to. With such fine-grained information, our framework enables more downstream potentials, such as object-level matching and pose estimation in a clustered environment. To get around the difficulty of label collection in the real wo…
Learnable keypoint detectors and descriptors are beginning to outperform classical hand-crafted feature extraction methods. Recent studies on self-supervised learning of visual representations have driven the increasing performance of learnable models based on deep networks. By leveraging traditional data augmentations and homography transformations, these networks learn to detect corners under adverse conditions such as extreme illumination changes. However, their generalization capabilities are limited to corner-like features detected a priori by classical methods or synthetically generated …
This paper introduces a new algorithm for unsupervised learning of keypoint detectors and descriptors, which demonstrates fast convergence and good performance across different datasets. The training procedure uses homographic transformation of images. The proposed model learns to detect points and generate descriptors on pairs of transformed images, which are easy for it to distinguish and repeatedly detect. The trained model follows SuperPoint architecture for ease of comparison, and demonstrates similar performance on natural images from HPatches dataset, and better performance on retina im…
Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions a…
Pattern Recognition Lab, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学模式识别实验室）；Department of Ophthalmology, Universitätsklinikum Erlangen（埃尔朗根大学医院眼科）
Ophthalmological imaging utilizes different imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography. Multiple images with different modalities or acquisition times are often analyzed for the diagnosis of retinal diseases. Automatically aligning the vessel structures in the images by means of multi-modal registration can support the ophthalmologists in their work. Our method uses a convolutional neural network to extract features of the vessel structure in multi-modal retinal images. We jointly train a keypoint detection an…
A.I. Labs, Alibaba Group（阿里巴巴集团人工智能实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
Researchers have attempted utilizing deep neural network (DNN) to learn novel local features from images inspired by its recent successes on a variety of vision tasks. However, existing DNN-based algorithms have not achieved such remarkable progress that could be partly attributed to insufficient utilization of the interactive characters between local feature detector and descriptor. To alleviate these difficulties, we emphasize two desired properties, i.e., repeatability and reliability, to simultaneously summarize the inherent and interactive characters of local feature detector and descript…
Pohang University of Science and Technology (POSTECH)（浦项科技大学）
Detecting robust keypoints from an image is an integral part of many computer vision problems, and the characteristic orientation and scale of keypoints play an important role for keypoint description and matching. Existing learning-based methods for keypoint detection rely on standard translation-equivariant CNNs but often fail to detect reliable keypoints against geometric variations. To learn to detect robust oriented keypoints, we introduce a self-supervised learning framework using rotation-equivariant CNNs. We propose a dense orientation alignment loss by an image pair generated by synth…
Stevens Institute of Technology（史蒂文斯理工学院）；Wormpex AI Research（Wormpex人工智能研究院）
In this paper, we propose an end-to-end framework that jointly learns keypoint detection, descriptor representation and cross-frame matching for the task of image-based 3D localization. Prior art has tackled each of these components individually, purportedly aiming to alleviate difficulties in effectively train a holistic network. We design a self-supervised image warping correspondence loss for both feature detection and matching, a weakly-supervised epipolar constraints loss on relative camera pose learning, and a directional matching scheme that detects key-point features in a source image …
In ophthalmological imaging, multiple imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography, are often involved to make a diagnosis of retinal disease. Multi-modal retinal registration techniques can assist ophthalmologists by providing a pixel-based comparison of aligned vessel structures in images from different modalities or acquisition times. To this end, we propose an end-to-end trainable deep learning method for multi-modal retinal image registration. Our method extracts convolutional features from the vessel struc…
In this paper, we present an omnidirectional localization and dense mapping system for a wide-baseline multiview stereo setup with ultra-wide field-of-view (FOV) fisheye cameras, which has a 360 degrees coverage of stereo observations of the environment. For more practical and accurate reconstruction, we first introduce improved and light-weighted deep neural networks for the omnidirectional depth estimation, which are faster and more accurate than the existing networks. Second, we integrate our omnidirectional depth estimates into the visual odometry (VO) and add a loop closing module for glo…
Renmin University of China（中国人民大学）；Visionary Intelligence Ltd（远见智能有限公司）；Tongren Hospital（同仁医院）
For retinal image matching (RIM), we propose SuperRetina, the first end-to-end method with jointly trainable keypoint detector and descriptor. SuperRetina is trained in a novel semi-supervised manner. A small set of (nearly 100) images are incompletely labeled and used to supervise the network to detect keypoints on the vascular tree. To attack the incompleteness of manual labeling, we propose Progressive Keypoint Expansion to enrich the keypoint labels at each training epoch. By utilizing a keypoint-based improved triplet loss as its description loss, SuperRetina produces highly discriminativ…
A number of computer vision tasks exploit a succinct representation of the visual content in the form of sets of local features. Given an input image, feature extraction algorithms identify a set of keypoints and assign to each of them a description vector, based on the characteristics of the visual content surrounding the interest point. Several tasks might require local features to be extracted from a video sequence, on a frame-by-frame basis. Although temporal downsampling has been proven to be an effective solution for mobile augmented reality and visual search, high temporal resolution is…
Mody University of Science and Technology（莫迪科技大学）；Moradabad Institute of Technology（莫拉达巴德理工学院）
Copy move forgery detection in digital images has become a very popular research topic in the area of image forensics. Due to the availability of sophisticated image editing tools and ever increasing hardware capabilities, it has become an easy task to manipulate the digital images. Passive forgery detection techniques are more relevant as they can be applied without the prior information about the image in question. Block based techniques are used to detect copy move forgery, but have limitations of large time complexity and sensitivity against affine operations like rotation and scaling. Key…
We propose a new method for estimating the relative pose between two images, where we jointly learn keypoint detection, description extraction, matching and robust pose estimation. While our architecture follows the traditional pipeline for pose estimation from geometric computer vision, all steps are learnt in an end-to-end fashion, including feature matching. We demonstrate our method for the task of visual localization of a query image within a database of images with known pose. Pairwise pose estimation has many practical applications for robotic mapping, navigation, and AR. For example, t…
University of Oxford（牛津大学）；National University of Defense Technology（国防科技大学）；Zhejiang University（浙江大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；University of Edinburgh（爱丁堡大学）；Durham University（杜伦大学）；Wuhan University（武汉大学）
Accurately describing and detecting 2D and 3D keypoints is crucial to establishing correspondences across images and point clouds. Despite a plethora of learning-based 2D or 3D local feature descriptors and detectors having been proposed, the derivation of a shared descriptor and joint keypoint detector that directly matches pixels and points remains under-explored by the community. This work takes the initiative to establish fine-grained correspondences between 2D images and 3D point clouds. In order to directly match pixels and points, a dual fully convolutional framework is presented that m…
Detecting image correspondences by feature matching forms the basis of numerous computer vision applications. Several detectors and descriptors have been presented in the past, addressing the efficient generation of features from interest points (keypoints) in an image. In this paper, we investigate eight binary descriptors (AKAZE, BoostDesc, BRIEF, BRISK, FREAK, LATCH, LUCID, and ORB) and eight interest point detector (AGAST, AKAZE, BRISK, FAST, HarrisLapalce, KAZE, ORB, and StarDetector). We have decoupled the detection and description phase to analyze the interest point detectors and then e…
Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Shenzhen Zhuke Innovation Technology (Altizure)（深圳珠科创新技术（Altizure））
Most existing studies on learning local features focus on the patch-based descriptions of individual keypoints, whereas neglecting the spatial relations established from their keypoint locations. In this paper, we go beyond the local detail representation by introducing context awareness to augment off-the-shelf local feature descriptors. Specifically, we propose a unified learning framework that leverages and aggregates the cross-modality contextual information, including (i) visual context from high-level image representation, and (ii) geometric context from 2D keypoint distribution. Moreove…
This study attempts to provide explanations, descriptions and evaluations of some most popular and current combinations of description and descriptor frameworks, namely SIFT, SURF, MSER, and BRISK for keypoint extractors and SIFT, SURF, BRISK, and FREAK for descriptors. Evaluations are made based on the number of matches of keypoints and repeatability in various image variations. It is used as the main parameter to assess how well combinations of algorithms are in matching objects with different variations. There are many papers that describe the comparison of detection and description feature…
High-resolution fingerprint recognition often relies on sophisticated matching algorithms based on hand-crafted keypoint descriptors, with pores being the most common keypoint choice. Our method is the opposite of the prevalent approach: we use instead a simple matching algorithm based on robust local pore descriptors that are learned from the data using a CNN. In order to train this CNN in a fully supervised manner, we describe how the automatic alignment of fingerprint images can be used to obtain the required training annotations, which are otherwise missing in all publicly available datase…
3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Meituan Inc.（美团公司）
Sapienza University（罗马第一大学）；University of Udine（乌迪内大学）；University of Bristol（布里斯托大学）
Estimating the 3D pose of a hand from a 2D image is a well-studied problem and a requirement for several real-life applications such as virtual reality, augmented reality, and hand gesture recognition. Currently, reasonable estimations can be computed from single RGB images, especially when a multi-task learning approach is used to force the system to consider the shape of the hand when its pose is determined. However, depending on the method used to represent the hand, the performance can drop considerably in real-life tasks, suggesting that stable descriptions are required to achieve satisfa…
MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching
National Engineering Research Center for Multimedia Software（多媒体软件国家工程研究中心）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机科学学院）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（湖北省多媒体与网络通信工程重点实验室）；Lee Kong Chian School of Medicine（李光耀医学院）；Nanyang Technological University（南洋理工大学）；Ningbo Institute of Materials Technology and Engineering（宁波材料技术与工程研究所）；Chinese Academy of Sciences（中国科学院）；Institute for Infocomm Research (I 2 R)（信息与通信研究所以（I 2 R））；Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR））
Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, China（人工智能研究院，深圳MSU-BIT大学，中国）；College of Mechatronics and Control Engineering, Shenzhen University（机电工程学院，深圳大学）；College of Computer Science and Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）
Feature detectors and descriptors have been successfully used for various computer vision tasks, such as video object tracking and content-based image retrieval. Many methods use image gradients in different stages of the detection-description pipeline to describe local image structures. Recently, some, or all, of these stages have been replaced by convolutional neural networks (CNNs), in order to increase their performance. A detector is defined as a selection problem, which makes it more challenging to implement as a CNN. They are therefore generally defined as regressors, converting input i…
ENS（巴黎高等师范学院）；Inria（法国国家信息与自动化研究所）；ETH Zurich（苏黎世联邦理工学院）；CTU in Prague（布拉格捷克理工大学）；Microsoft（微软）；Tokyo Institute of Technology（东京工业大学）；Chalmers University of Technology（查尔姆斯理工大学）
In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions. We propose an approach where a single convolutional neural network plays a dual role: It is simultaneously a dense feature descriptor and a feature detector. By postponing the detection to a later stage, the obtained keypoints are more stable than their traditional counterparts based on early detection of low-level structures. We show that this model can be trained using pixel correspondences extracted from readily available large-scale SfM reconstructions, without any furthe…
Boosting Semi-Supervised 2D Human Pose Estimation by Revisiting Data Augmentation and Consistency Training
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Chongqing Academy of Science and Technology（重庆科学技术研究院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）
A successful point cloud registration often lies on robust establishment of sparse matches through discriminative 3D local features. Despite the fast evolution of learning-based 3D feature descriptors, little attention has been drawn to the learning of 3D feature detectors, even less for a joint learning of the two tasks. In this paper, we leverage a 3D fully convolutional network for 3D point clouds, and propose a novel and practical learning mechanism that densely predicts both a detection score and a description feature for each 3D point. In particular, we propose a keypoint selection strat…
Tecnalia Research and Innovation（泰克纳利亚研究与创新中心）；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））
Detection and description of keypoints from an image is a well-studied problem in Computer Vision. Some methods like SIFT, SURF or ORB are computationally really efficient. This paper proposes a solution for a particular case study on object recognition of industrial parts based on hierarchical classification. Reducing the number of instances leads to better performance, indeed, that is what the use of the hierarchical classification is looking for. We demonstrate that this method performs better than using just one method like ORB, SIFT or FREAK, despite being fairly slower.
