In the thesis we consider the problem of local feature descriptor learning for wide baseline stereo focusing on the HardNet descriptor, which is close to state-of-the-art. AMOS Patches dataset is introduced, which improves robustness to illumination and appearance changes. It is based on registered images from selected cameras from the AMOS dataset. We provide recommendations on the patch dataset creation process and evaluate HardNet trained on data of different modalities. We also introduce a dataset combination and reduction methods, that allow comparable performance on a significantly smaller dataset.
HardNet8, consistently outperforming the original HardNet, benefits from the architectural choices made: connectivity pattern, final pooling, receptive field, CNN building blocks found by manual or automatic search algorithms -- DARTS. We show impact of overlooked hyperparameters such as batch size and length of training on the descriptor quality. PCA dimensionality reduction further boosts performance and also reduces memory footprint.
Finally, the insights gained lead to two HardNet8 descriptors: one performing well on a variety of benchmarks -- HPatches, AMOS Patches and IMW Phototourism, the other is optimized for IMW Phototourism.
Deep neural networks have demonstrated state-of-the-art performance for feature-based image matching through the advent of new large and diverse datasets. However, there has been little work on evaluating the computational cost, model size, and matching accuracy tradeoffs for these models. This paper explicitly addresses these practical metrics by considering the state-of-the-art HardNet model. We observe a significant redundancy in the learned weights, which we exploit through the use of depthwise separable layers and an efficient Tucker decomposition. We demonstrate that a combination of the…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Mercedes Benz Research and Development India（梅赛德斯-奔驰印度研发中心）
We propose a new dataset for learning local image descriptors which can be used for significantly improved patch matching. Our proposed dataset consists of an order of magnitude more number of scenes, images, and positive and negative correspondences compared to the currently available Multi-View Stereo (MVS) dataset from Brown et al. The new dataset also has better coverage of the overall viewpoint, scale, and lighting changes in comparison to the MVS dataset. Our dataset also provides supplementary information like RGB patches with scale and rotations values, and intrinsic and extrinsic came…
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）；Instituto Federal do Sul de Minas Gerais（南米纳斯吉拉斯联邦学院）；Pontificia Universidade Catolica do Paraná（巴拉那天主教大学）
Information from an image occurs over multiple and distinct spatial scales. Image pyramid multiresolution representations are a useful data structure for image analysis and manipulation over a spectrum of spatial scales. This paper employs the Gaussian-Laplacian pyramid to treat different spatial frequency bands of a texture separately. First, we generate three images corresponding to three levels of the Gaussian-Laplacian pyramid for an input image to capture intrinsic details. Then we aggregate features extracted from gray and color texture images using bio-inspired texture descriptors, info…
ETH Zürich（苏黎世联邦理工学院）；University of Bonn（波恩大学）；Lund University（隆德大学）；Microsoft（微软）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
A distinctive representation of image patches in form of features is a key component of many computer vision and robotics tasks, such as image matching, image retrieval, and visual localization. State-of-the-art descriptors, from hand-crafted descriptors such as SIFT to learned ones such as HardNet, are usually high dimensional; 128 dimensions or even more. The higher the dimensionality, the larger the memory consumption and computational time for approaches using such descriptors. In this paper, we investigate multi-layer perceptrons (MLPs) to extract low-dimensional but high-quality descript…
Chongqing University of Posts and Telecommunications（重庆邮电大学）
Local binary pattern (LBP) as a kind of local feature has shown its simplicity, easy implementation and strong discriminating power in image recognition. Although some LBP variants are specifically investigated for color image recognition, the color information of images is not adequately considered and the curse of dimensionality in classification is easily caused in these methods. In this paper, a color-related local binary pattern (cLBP) which learns the dominant patterns from the decoded LBP is proposed for color images recognition. This paper first proposes a relative similarity space (RS…
We propose a self-supervised learning framework that uses unlabeled monocular video sequences to generate large-scale supervision for training a Visual Odometry (VO) frontend, a network which computes pointwise data associations across images. Our self-improving method enables a VO frontend to learn over time, unlike other VO and SLAM systems which require time-consuming hand-tuning or expensive data collection to adapt to new environments. Our proposed frontend operates on monocular images and consists of a single multi-task convolutional neural network which outputs 2D keypoints locations, k…
Yandex（Yandex（央捷科斯））；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
The top-performing systems for billion-scale high-dimensional approximate nearest neighbor (ANN) search are all based on two-layer architectures that include an indexing structure and a compressed datapoints layer. An indexing structure is crucial as it allows to avoid exhaustive search, while the lossy data compression is needed to fit the dataset into RAM. Several of the most successful systems use product quantization (PQ) for both the indexing and the dataset compression layers. These systems are however limited in the way they exploit the interaction of product quantization processes that…
QUT Centre for Robotics（昆士兰科技大学机器人中心）；School of Electrical Engineering and Robotics, QUT（昆士兰科技大学电气工程与机器人学院）；Queensland University of Technology (QUT)（昆士兰科技大学）
Visual Place Recognition (VPR) is the task of matching current visual imagery from a camera to images stored in a reference map of the environment. While initial VPR systems used simple direct image methods or hand-crafted visual features, recent work has focused on learning more powerful visual features and further improving performance through either some form of sequential matcher / filter or a hierarchical matching process. In both cases the performance of the initial single-image based system is still far from perfect, putting significant pressure on the sequence matching or (in the case …
CTU in Prague（布拉格捷克理工大学）；Facebook AI Research（脸书人工智能研究院）
We propose a multiple-kernel local-patch descriptor based on efficient match kernels from pixel gradients. It combines two parametrizations of gradient position and direction, each parametrization provides robustness to a different type of patch mis-registration: polar parametrization for noise in the patch dominant orientation detection, Cartesian for imprecise location of the feature point. Combined with whitening of the descriptor space, that is learned with or without supervision, the performance is significantly improved. We analyze the effect of the whitening on patch similarity and demo…
ConKeD++ -- Improving descriptor learning for retinal image registration: A comprehensive study of contrastive losses
Instituto de Investigacion Biomédica de A Coruña (INIBIC)（拉科鲁尼亚生物医学研究所）；Universidade da Coruña（拉科鲁尼亚大学）；Departamento de Ciencias de la Computación y Tecnologías de la Información（计算机科学与信息技术系）；Grupo VARPA（VARPA团队）
Insight Centre for Data Analytics, Dublin City University (DCU)（都柏林城市大学洞察数据分析中心（DCU））
This paper explores novel approaches for improving the spatial codification for the pooling of local descriptors to solve the semantic segmentation problem. We propose to partition the image into three regions for each object to be described: Figure, Border and Ground. This partition aims at minimizing the influence of the image context on the object description and vice versa by introducing an intermediate zone around the object contour. Furthermore, we also propose a richer visual descriptor of the object by applying a Spatial Pyramid over the Figure region. Two novel Spatial Pyramid configu…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医疗机构）
High-quality 3D reconstructions from endoscopy video play an important role in many clinical applications, including surgical navigation where they enable direct video-CT registration. While many methods exist for general multi-view 3D reconstruction, these methods often fail to deliver satisfactory performance on endoscopic video. Part of the reason is that local descriptors that establish pair-wise point correspondences, and thus drive reconstruction, struggle when confronted with the texture-scarce surface of anatomy. Learning-based dense descriptors usually have larger receptive fields ena…
Department of Mathematics, University of California Irvine（加州大学欧文分校数学系）
The main purpose of this paper is to propose a new preprocessing step in order to improve local feature descriptors and texture classification. Preprocessing is implemented by using transformations which help highlight salient features that play a significant role in texture recognition. We evaluate and compare four different competing methods: three different anisotropic diffusion methods including the classical anisotropic Perona-Malik diffusion and two subsequent regularizations of it and the application of a Gaussian kernel, which is the classical multiscale approach in texture analysis. T…
Universidad Politécnica de Madrid（马德里理工大学）；Centre for Automation and Robotics (CAR UPM-CSIC)（自动化与机器人中心（马德里理工大学-西班牙国家研究委员会））；ETSIDI（马德里理工大学工业设计工程学院）
The use of 3D cameras for gait analysis has been highly questioned due to the low accuracy they have demonstrated in the past. The objective of the study presented in this paper is to improve the accuracy of the estimations made by robot-mounted 3D cameras in human gait analysis by applying a supervised learning stage. The 3D camera was mounted in a mobile robot to obtain a longer walking distance. This study shows an improvement in detection of kinematic gait signals and gait descriptors by post-processing the raw estimations of the camera using artificial neural networks trained with the dat…
Improving Autism Detection with Multimodal Behavioral Analysis
Center for Cognitive Interaction Technology (CITEC), Bielefeld University（认知交互技术中心（CITEC），比勒菲尔德大学）；Institute of Psychology, Humboldt University of Berlin（心理学研究所，洪堡大学）；Department of Psychiatry and Psychotherapy, Medical Center-University of Freiburg（精神病学与心理治疗系，弗赖堡医学院-大学）
