关键点检测与匹配是许多计算机视觉问题中的一项基础任务，应用范围从形状重建、运动恢复结构（structure from motion），到 AR/VR 应用和机器人技术。这是一个已得到充分研究的问题，并取得了显著成功，例如 SIFT 以及较新的深度学习方法。尽管这些技术在噪声、光照变化和刚体运动变换方面表现出很强的鲁棒性，但人们对图像畸变敏感性的关注较少。在本文中，我们关注由用于图像采集的相机几何所导致的情况，并考虑鱼眼图像与投影（透视）图像这一混合场景之间的关键点检测和匹配问题。我们以一种最先进的方法为基础，推导出一种自监督流程，使其能够训练兴趣点检测器和描述符网络。我们还收集了两个新的数据集，用于在这一尚未探索的场景中进行额外训练和测试；我们证明，现有方法并不理想，因为它们是为传统投影条件设计的，而所提出的方法被证明是最有效的。
英文摘要：
Keypoint detection and matching is a fundamental task in many computer vision problems, from shape reconstruction, to structure from motion, to AR/VR applications and robotics. It is a well-studied problem with remarkable successes such as SIFT, and more recent deep learning approaches. While great robustness is exhibited by these techniques with respect to noise, illumination variation, and rigid motion transformations, less attention has been placed on image distortion sensitivity. In this work, we focus on the case when this is caused by the geometry of the cameras used for image acquisition, and consider the keypoint detection and matching problem between the hybrid scenario of a fisheye and a projective image. We build on a state-of-the-art approach and derive a self-supervised procedure that enables training an interest point detector and descriptor network. We also collected two new datasets for additional training and testing in this unexplored scenario, and we demonstrate that current approaches are suboptimal because they are designed to work in traditional projective conditions, while the proposed approach turns out to be the most effective.
Valeo DAR Kronach（法雷奥DAR克罗纳赫）；Technische Universität Ilmenau（伊尔默瑙工业大学）；Technische Universität Braunschweig（布伦瑞克工业大学）；Valeo Vision Systems（法雷奥视觉系统）
State-of-the-art self-supervised learning approaches for monocular depth estimation usually suffer from scale ambiguity. They do not generalize well when applied on distance estimation for complex projection models such as in fisheye and omnidirectional cameras. This paper introduces a novel multi-task learning strategy to improve self-supervised monocular distance estimation on fisheye and pinhole camera images. Our contribution to this work is threefold: Firstly, we introduce a novel distance estimation network architecture using a self-attention based encoder coupled with robust semantic fe…
OPPO US Research Center（OPPO美国研究中心）；InnoPeak Technology, Inc.（InnoPeak技术公司）；Wing LLC（Wing有限责任公司）
In this paper, we deal with the problem of monocular depth estimation for fisheye cameras in a self-supervised manner. A known issue of self-supervised depth estimation is that it suffers in low-light/over-exposure conditions and in large homogeneous regions. To tackle this issue, we propose a novel ordinal distillation loss that distills the ordinal information from a large teacher model. Such a teacher model, since having been trained on a large amount of diverse data, can capture the depth ordering information well, but lacks in preserving accurate scene geometry. Combined with self-supervi…
National Chiao Tung University（国立交通大学）；Chunghwa Telecom Laboratories（中华电信实验室）
Pedestrian detection in images is a topic that has been studied extensively, but existing detectors designed for perspective images do not perform as successfully on images taken with top-view fisheye cameras, mainly due to the orientation variation of people in such images. In our proposed approach, several perspective views are generated from a fisheye image and then concatenated to form a composite image. As pedestrians in this composite image are more likely to be upright, existing detectors designed and trained for perspective images can be applied directly without additional training. We…
Valeo DAR Kronach（法雷奥DAR克罗纳赫）；TU Ilmenau（伊尔梅瑙工业大学）；TU Braunschweig（布伦瑞克工业大学）；Valeo Vision Systems（法雷奥视觉系统）
A 360° perception of scene geometry is essential for automated driving, notably for parking and urban driving scenarios. Typically, it is achieved using surround-view fisheye cameras, focusing on the near-field area around the vehicle. The majority of current depth estimation approaches focus on employing just a single camera, which cannot be straightforwardly generalized to multiple cameras. The depth estimation model must be tested on a variety of cameras equipped to millions of cars with varying camera geometries. Even within a single car, intrinsics vary due to manufacturing tolerances. De…
Valeo DAR Kronach（法雷奥达尔斯克罗纳赫）；ENSTA ParisTech Palaiseau（巴黎高等理工学院帕莱索校区）；Valeo Vision Systems（法雷奥视觉系统公司）；Technische Universität Ilmenau（伊尔梅瑙工业大学）
Fisheye cameras are commonly used in applications like autonomous driving and surveillance to provide a large field of view ($>180^{\circ}$). However, they come at the cost of strong non-linear distortions which require more complex algorithms. In this paper, we explore Euclidean distance estimation on fisheye cameras for automotive scenes. Obtaining accurate and dense depth supervision is difficult in practice, but self-supervised learning approaches show promising results and could potentially overcome the problem. We present a novel self-supervised scale-aware framework for learning Euclide…
Maynooth University（梅努斯大学）；University of Limerick（利默里克大学）；Valeo Vision Systems（法雷奥视觉系统公司）
Keypoint detection and description is a commonly used building block in computer vision systems particularly for robotics and autonomous driving. However, the majority of techniques to date have focused on standard cameras with little consideration given to fisheye cameras which are commonly used in urban driving and automated parking. In this paper, we propose a novel training and evaluation pipeline for fisheye images. We make use of SuperPoint as our baseline which is a self-supervised keypoint detector and descriptor that has achieved state-of-the-art results on homography estimation. We i…
SimFIR: A Simple Framework for Fisheye Image Rectification with Self-supervised Representation Learning
University of Science and Technology of China（中国科学技术大学）；Zhangjiang Laboratory（张江实验室）；The University of Sydney（悉尼大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
提出 SimFIR 自监督框架，利用 ViT 图像块表示与统一畸变感知预训练任务学习细粒度畸变特征，从而提升鱼眼图像校正性能和真实场景泛化能力。
Adapting CNNs for Fisheye Cameras without Retraining
Australian Centre For Robotics (ACFR)（澳大利亚机器人中心）；School of Aerospace, Mechanical and Mechatronic Engineering（航空航天、机械与机电工程学院）；The University of Sydney（悉尼大学）
In this report, we proposed a 3D reconstruction method for the full-view fisheye camera. The camera we used is Ricoh Theta, which captures spherical images and has a wide field of view (FOV). The conventional stereo apporach based on perspective camera model cannot be directly applied and instead we used a spherical camera model to depict the relation between 3D point and its corresponding observation in the image. We implemented a system that can reconstruct the 3D scene using captures from two or more cameras. A GUI is also created to allow users to control the view perspective and obtain a …
General Motors Advanced Technical Center Israel（通用汽车以色列高级技术中心）
Existing monocular 3D object detection methods have been demonstrated on rectilinear perspective images and fail in images with alternative projections such as those acquired by fisheye cameras. Previous works on object detection in fisheye images have focused on 2D object detection, partly due to the lack of 3D datasets of such images. In this work, we show how to use existing monocular 3D object detection models, trained only on rectilinear images, to detect 3D objects in images from fisheye cameras, without using any fisheye training data. We outperform the only existing method for monocula…
Fast binary descriptors build the core for many vision based applications with real-time demands like object detection, Visual Odometry or SLAM. Commonly it is assumed, that the acquired images and thus the patches extracted around keypoints originate from a perspective projection ignoring image distortion or completely different types of projections such as omnidirectional or fisheye. Usually the deviations from a perfect perspective projection are corrected by undistortion. Latter, however, introduces severe artifacts if the cameras field-of-view gets larger. In this paper, we propose a dist…
Friedrich-Alexander University Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Although fisheye cameras are in high demand in many application areas due to their large field of view, many image and video signal processing tasks such as motion compensation suffer from the introduced strong radial distortions. A recently proposed projection-based approach takes the fisheye projection into account to improve fisheye motion compensation. However, the approach does not consider the large field of view of fisheye lenses that requires the consideration of different motion planes in 3D space. We propose a novel viewport-adaptive motion compensation technique that applies the mot…
Thales（泰雷兹集团）；Thales Digital Solutions（泰雷兹数字解决方案公司）；INRIA（法国国家信息与自动化研究所）
Conventional convolution neural networks (CNNs) trained on narrow Field-of-View (FoV) images are the state-of-the-art approaches for object recognition tasks. Some methods proposed the adaptation of CNNs to ultra-wide FoV images by learning deformable kernels. However, they are limited by the Euclidean geometry and their accuracy degrades under strong distortions caused by fisheye projections. In this work, we demonstrate that learning the shape of convolution kernels in non-Euclidean spaces is better than existing deformable kernel methods. In particular, we propose a new approach that learns…
Signal Processing Friedrich-Alexander University Erlangen-N\"urnberg (FAU), Cauerstr. 7, 91058 Erlangen, Germany
In video surveillance as well as automotive applications, so-called fisheye cameras are often employed to capture a very wide angle of view. As such cameras depend on projections quite different from the classical perspective projection, the resulting fisheye image and video data correspondingly exhibits non-rectilinear image characteristics. Typical image and video processing algorithms, however, are not designed for these fisheye characteristics. To be able to develop and evaluate algorithms specifically adapted to fisheye images and videos, a corresponding test data set is therefore introdu…
Location-guided Head Pose Estimation for Fisheye Image
School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院）；Data Science Research Center, Duke Kunshan University（昆山杜克大学数据科学研究中心）；Brigham Young University（杨百翰大学）
FishEye8K: A Benchmark and Dataset for Fisheye Camera Object Detection
United Arab Emirates University（阿联酋大学）；University at Albany — State University of New York（纽约州立大学奥尔巴尼分校）；National Yang Ming Chiao Tung University（国立阳明交通大学）；Mongolian University of Science and Technology（蒙古科技大学）；National Center for High-Performance Computing（国家高速网络与计算中心）
本文构建开放的 FishEye8K 鱼眼道路目标检测数据集，含 8,000 张图像、157K 个边界框，并评估多种 YOLO 系列 SoTA 模型。
Stanford AI Lab（斯坦福人工智能实验室）；Stanford University（斯坦福大学）
We present semi-supervised deep learning approaches for traversability estimation from fisheye images. Our method, GONet, and the proposed extensions leverage Generative Adversarial Networks (GANs) to effectively predict whether the area seen in the input image(s) is safe for a robot to traverse. These methods are trained with many positive images of traversable places, but just a small set of negative images depicting blocked and unsafe areas. This makes the proposed methods practical. Positive examples can be collected easily by simply operating a robot through traversable spaces, while obta…
Signal Processing Friedrich-Alexander University Erlangen-N\"urnberg (FAU), Cauerstr. 7, 91058 Erlangen, Germany
Capturing large fields of view with only one camera is an important aspect in surveillance and automotive applications, but the wide-angle fisheye imagery thus obtained exhibits very special characteristics that may not be very well suited for typical image and video processing methods such as motion estimation. This paper introduces a motion estimation method that adapts to the typical radial characteristics of fisheye video sequences by making use of an equisolid re-projection after moving part of the motion vector search into the perspective domain via a corresponding back-projection. By co…
In this paper, we present a multi-camera visual odometry (VO) system for an autonomous vehicle. Our system mainly consists of a virtual LiDAR and a pose tracker. We use a perspective transformation method to synthesize a surround-view image from undistorted fisheye camera images. With a semantic segmentation model, the free space can be extracted. The scans of the virtual LiDAR are generated by discretizing the contours of the free space. As for the pose tracker, we propose a visual odometry system fusing both the feature matching and the virtual LiDAR scan matching results. Only those feature…
We propose a novel 3D human pose detector using two panoramic cameras. We show that transforming fisheye perspectives to rectilinear views allows a direct application of two-dimensional deep-learning pose estimation methods, without the explicit need for a costly re-training step to compensate for fisheye image distortions. By utilizing panoramic cameras, our method is capable of accurately estimating human poses over a large field of view. This renders our method suitable for ergonomic analyses and other pose based assessments.
The University of Sydney（悉尼大学）；UBTECH Sydney AI Centre（优必选悉尼人工智能中心）
Deep learning has demonstrated its power in image rectification by leveraging the representation capacity of deep neural networks via supervised training based on a large-scale synthetic dataset. However, the model may overfit the synthetic images and generalize not well on real-world fisheye images due to the limited universality of a specific distortion model and the lack of explicitly modeling the distortion and rectification process. In this paper, we propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted image…
Self-supervised monocular depth estimation has been widely investigated to estimate depth images and relative poses from RGB images. This framework is attractive for researchers because the depth and pose networks can be trained from just time sequence images without the need for the ground truth depth and poses. In this work, we estimate the depth around a robot (360 degree view) using time sequence spherical camera images, from a camera whose parameters are unknown. We propose a learnable axisymmetric camera model which accepts distorted spherical camera images with two fisheye camera images…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Toyota Research Institute（丰田研究院）
Camera calibration is integral to robotics and computer vision algorithms that seek to infer geometric properties of the scene from visual input streams. In practice, calibration is a laborious procedure requiring specialized data collection and careful tuning. This process must be repeated whenever the parameters of the camera change, which can be a frequent occurrence for mobile robots and autonomous vehicles. In contrast, self-supervised depth and ego-motion estimation approaches can bypass explicit calibration by inferring per-frame projection models that optimize a view synthesis objectiv…
Toyota Research Institute（丰田研究院）；Toyota Technological Institute at Chicago（芝加哥丰田技术学院）
Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model -- usually the standard pinhole geometry -- leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the c…
Chungnam National University（忠南大学）；Electronics and Telecommunication Research Institute（电子与电信研究院）
Recently, there has been growing attention on an end-to-end deep learning-based stitching model. However, the most challenging point in deep learning-based stitching is to obtain pairs of input images with a narrow field of view and ground truth images with a wide field of view captured from real-world scenes. To overcome this difficulty, we develop a weakly-supervised learning mechanism to train the stitching model without requiring genuine ground truth images. In addition, we propose a stitching model that takes multiple real-world fisheye images as inputs and creates a 360 output image in a…
Chemnitz University of Technology（开姆尼茨工业大学）；Faculty of Electrical Engineering and Information Technology（电气工程与信息技术学院）
Recent work about synthetic indoor datasets from perspective views has shown significant improvements of object detection results with Convolutional Neural Networks(CNNs). In this paper, we introduce THEODORE: a novel, large-scale indoor dataset containing 100,000 high-resolution diversified fisheye images with 14 classes. To this end, we create 3D virtual environments of living rooms, different human characters and interior textures. Beside capturing fisheye images from virtual environments we create annotations for semantic segmentation, instance masks and bounding boxes for object detection…
Zhejiang University（浙江大学）；The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
The pretrain-finetune paradigm is a classical pipeline in visual learning. Recent progress on unsupervised pretraining methods shows superior transfer performance to their supervised counterparts. This paper revisits this phenomenon and sheds new light on understanding the transferability gap between unsupervised and supervised pretraining from a multilayer perceptron (MLP) perspective. While previous works focus on the effectiveness of MLP on unsupervised image classification where pretraining and evaluation are conducted on the same dataset, we reveal that the MLP projector is also the key f…
iRobot（iRobot公司）；University College London（伦敦大学学院）
Representation learning approaches typically rely on images of objects captured from a single perspective that are transformed using affine transformations. Additionally, self-supervised learning, a successful paradigm of representation learning, relies on instance discrimination and self-augmentations which cannot always bridge the gap between observations of the same object viewed from a different perspective. Viewing an object from multiple perspectives aids holistic understanding of an object which is particularly important in situations where data annotations are limited. In this paper, w…
Dual-Perspective Knowledge Enrichment for Semi-Supervised 3D Object Detection
Nanyang Technological University（南洋理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Hyundai Motor Group Innovation Center in Singapore（现代汽车集团新加坡创新中心）
Boosting Semi-Supervised 2D Human Pose Estimation by Revisiting Data Augmentation and Consistency Training
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Chongqing Academy of Science and Technology（重庆科学技术研究院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Human interpretation of the world encompasses the use of symbols to categorize sensory inputs and compose them in a hierarchical manner. One of the long-term objectives of Computer Vision and Artificial Intelligence is to endow machines with the capacity of structuring and interpreting the world as we do. Towards this goal, recent methods have successfully been able to decompose and disentangle video sequences into their composing objects and dynamics, in a self-supervised fashion. However, there has been a scarce effort in giving interpretation to the dynamics of the scene. We propose a metho…
Modern deep learning-based 3D pose estimation approaches require plenty of 3D pose annotations. However, existing 3D datasets lack diversity, which limits the performance of current methods and their generalization ability. Although existing methods utilize 2D pose annotations to help 3D pose estimation, they mainly focus on extracting 2D structural constraints from 2D poses, ignoring the 3D information hidden in the images. In this paper, we propose a novel method to extract weak 3D information directly from 2D images without 3D pose supervision. Firstly, we utilize 2D pose annotations and pe…
Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University（北京交通大学交通数据分析与挖掘北京市重点实验室）
This paper addresses semi-supervised semantic segmentation by exploiting a small set of images with pixel-level annotations (strong supervisions) and a large set of images with only image-level annotations (weak supervisions). Most existing approaches aim to generate accurate pixel-level labels from weak supervisions. However, we observe that those generated labels still inevitably contain noisy labels. Motivated by this observation, we present a novel perspective and formulate this task as a problem of learning with pixel-level label noise. Existing noisy label methods, nevertheless, mainly a…
Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired b…
Deep learning-based underwater object detection (UOD) remains a major challenge due to the degraded visibility and difficulty to obtain sufficient underwater object images captured from various perspectives for training. To address these issues, this paper presents a high-precision UOD based on joint self-supervised deblurring and improved spatial transformer network. A self-supervised deblurring subnetwork is introduced into the designed multi-task learning aided object detection architecture to force the shared feature extraction module to output clean features for detection subnetwork. Aimi…
Diversity-enhanced Collaborative Mamba for Semi-supervised Medical Image Segmentation
School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院、南京大学）；School of Electrical and Information Engineering, The University of Sydney（电气与信息工程学院、悉尼大学）
One of the greatest obstacles in the adoption of deep neural networks for new applications is that training the network typically requires a large number of manually labeled training samples. We empirically investigate the scenario where one has access to large amounts of unlabeled data but require labeling only a single prototypical sample per class in order to train a deep network (i.e., one-shot semi-supervised learning). Specifically, we investigate the recent results reported in FixMatch for one-shot semi-supervised learning to understand the factors that affect and impede high accuracies…
NEC Laboratories America（NEC美国实验室）；UC San Diego（加州大学圣地亚哥分校）
We propose an end-to-end network that takes a single perspective RGB image of a complex road scene as input, to produce occlusion-reasoned layouts in perspective space as well as a parametric bird's-eye-view (BEV) space. In contrast to prior works that require dense supervision such as semantic labels in perspective view, our method only requires human annotations for parametric attributes that are cheaper and less ambiguous to obtain. To solve this challenging task, our design is comprised of modules that incorporate inductive biases to learn occlusion-reasoning, geometric transformation and …
In recent years, supervised learning has been widely used in various tasks of optical remote sensing image understanding, including remote sensing image classification, pixel-wise segmentation, change detection, and object detection. The methods based on supervised learning need a large amount of high-quality training data and their performance highly depends on the quality of the labels. However, in practical remote sensing applications, it is often expensive and time-consuming to obtain large-scale data sets with high-quality labels, which leads to a lack of sufficient supervised information…
Modern video object segmentation (VOS) algorithms have achieved remarkably high performance in a sequential processing order, while most of currently prevailing pipelines still show some obvious inadequacy like accumulative error, unknown robustness or lack of proper interpretation tools. In this paper, we place the semi-supervised video object segmentation problem into a cyclic workflow and find the defects above can be collectively addressed via the inherent cyclic property of semi-supervised VOS systems. Firstly, a cyclic mechanism incorporated to the standard sequential flow can produce mo…
Rethinking Pseudo-Label Guided Learning for Weakly Supervised Temporal Action Localization from the Perspective of Noise Correction
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）
Rethinking Generalizability and Discriminability of Self-Supervised Learning from Evolutionary Game Theory Perspective
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
Timestamp-Supervised Action Segmentation from the Perspective of Clustering
Institute of Software, Chinese Academy of Science（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）
