The ubiquitous multi-camera setup on modern autonomous vehicles provides an opportunity to construct surround-view depth. Existing methods, however, either perform independent monocular depth estimations on each camera or rely on computationally heavy self attention mechanisms. In this paper, we propose a novel guided attention architecture, EGA-Depth, which can improve both the efficiency and accuracy of self-supervised multi-camera depth estimation. More specifically, for each camera, we use its perspective view as the query to cross-reference its neighboring views to derive informative features for this camera view. This allows the model to perform attention only across views with considerable overlaps and avoid the costly computations of standard self-attention. Given its efficiency, EGA-Depth enables us to exploit higher-resolution visual features, leading to improved accuracy. Furthermore, EGA-Depth can incorporate more frames from previous time steps as it scales linearly w.r.t. the number of views and frames. Extensive experiments on two challenging autonomous driving benchmarks nuScenes and DDAD demonstrate the efficacy of our proposed EGA-Depth and show that it achieves the new state-of-the-art in self-supervised multi-camera depth estimation.
This work proposes a novel motion guided method for target-less self-calibration of a LiDAR and camera and use the re-projection of LiDAR points onto the image reference frame for real-time depth upsampling. The calibration parameters are estimated by optimizing an objective function that penalizes distances between 2D and re-projected 3D motion vectors obtained from time-synchronized image and point cloud sequences. For upsampling, a simple, yet effective and time efficient formulation that minimizes depth gradients subject to an equality constraint involving the LiDAR measurements is propose…
SDGE: Stereo Guided Depth Estimation for 360$^\circ$ Camera Sets
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；SZ DJI Technology Co., Ltd.（深圳市大疆创新科技有限公司）；The University of New South Wales（新南威尔士大学）
Nanjing University of Science and Technology（南京理工大学）；University of Macau（澳门大学）
Cross-spectrum depth estimation aims to provide a depth map in all illumination conditions with a pair of dual-spectrum images. It is valuable for autonomous vehicle applications when the vehicle is equipped with two cameras of different modalities. However, images captured by different-modality cameras can be photometrically quite different. Therefore, cross-spectrum depth estimation is a very challenging problem. Moreover, the shortage of large-scale open-source datasets also retards further research in this field. In this paper, we propose an unsupervised visible-light image guided cross-sp…
NEC Labs America（NEC美国实验室）；University of California San Diego（加州大学圣迭戈分校）
Learning methods for relative camera pose estimation have been developed largely in isolation from classical geometric approaches. The question of how to integrate predictions from deep neural networks (DNNs) and solutions from geometric solvers, such as the 5-point algorithm, has as yet remained under-explored. In this paper, we present a novel framework that involves probabilistic fusion between the two families of predictions during network training, with a view to leveraging their complementary benefits in a learnable way. The fusion is achieved by learning the DNN uncertainty under explic…
This work addresses cross-view camera pose estimation, i.e., determining the 3-Degrees-of-Freedom camera pose of a given ground-level image w.r.t. an aerial image of the local area. We propose SliceMatch, which consists of ground and aerial feature extractors, feature aggregators, and a pose predictor. The feature extractors extract dense features from the ground and aerial images. Given a set of candidate camera poses, the feature aggregators construct a single ground descriptor and a set of pose-dependent aerial descriptors. Notably, our novel aerial feature aggregator has a cross-view atten…
EGP3D: Edge-guided Geometric Preserving 3D Point Cloud Super-resolution for RGB-D camera
School of Data Science, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）数据科学学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；City University of Hong Kong（香港城市大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Tsinghua University（清华大学）
University at Buffalo（布法罗大学）；Amazon（亚马逊）；Amazon Prime Video（亚马逊Prime Video）
Existing approaches for Structure from Motion (SfM) produce impressive 3-D reconstruction results especially when using imagery captured with large parallax. However, to create engaging video-content in movies and TV shows, the amount by which a camera can be moved while filming a particular shot is often limited. The resulting small-motion parallax between video frames makes standard geometry-based SfM approaches not as effective for movies and TV shows. To address this challenge, we propose a simple yet effective approach that uses single-frame depth-prior obtained from a pretrained network …
Camera pose estimation or camera relocalization is the centerpiece in numerous computer vision tasks such as visual odometry, structure from motion (SfM) and SLAM. In this paper we propose a neural network approach with a graph transformer backbone, namely TransCamP, to address the camera relocalization problem. In contrast with prior work where the pose regression is mainly guided by photometric consistency, TransCamP effectively fuses the image features, camera pose information and inter-frame relative camera motions into encoded graph attributes and is trained towards the graph consistency …
Imperial College London（帝国理工学院）；Facebook Reality Labs（Facebook现实实验室）
In this paper, we address the problem of camera pose estimation in outdoor and indoor scenarios. In comparison to the currently top-performing methods that rely on 2D to 3D matching, we propose a model that can directly regress the camera pose from images with significantly higher accuracy than existing methods of the same class. We first analyse why regression methods are still behind the state-of-the-art, and we bridge the performance gap with our new approach. Specifically, we propose a way to overcome the biased training data by a novel training technique, which generates poses guided by a…
Entropy-Guided Agreement-Diversity: A Semi-Supervised Active Learning Framework for Fetal Head Segmentation in Ultrasound
Taighde \' E ireann – Research Ireland Centre for Research Training in Machine Learning, Ireland School of Computer Science, University College Dublin, Ireland School of Medicine, University College Dublin, Ireland
Event-guided 3D Gaussian Splatting for Dynamic Human and Scene Reconstruction
College of Optical Science and Engineering, Zhejiang University, China（浙江大学光学工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；School of Artificial Intelligence and Robotics, Hunan University, China（湖南大学人工智能与机器人学院）
We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dyn…
RGB-D cameras, which give an RGB image to- gether with depths, are becoming increasingly popular for robotic perception. In this paper, we address the task of detecting commonly found objects in the 3D point cloud of indoor scenes obtained from such cameras. Our method uses a graphical model that captures various features and contextual relations, including the local visual appearance and shape cues, object co-occurence relationships and geometric relationships. With a large number of object classes and relations, the model's parsimony becomes important and we address that by using multiple ty…
State Key Laboratory of Computer Science Institute of Software Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Machine Intelligence Technology Lab Alibaba Group（阿里巴巴集团机器智能技术实验室）
Multi-Target Multi-Camera Tracking has a wide range of applications and is the basis for many advanced inferences and predictions. This paper describes our solution to the Track 3 multi-camera vehicle tracking task in 2021 AI City Challenge (AICITY21). This paper proposes a multi-target multi-camera vehicle tracking framework guided by the crossroad zones. The framework includes: (1) Use mature detection and vehicle re-identification models to extract targets and appearance features. (2) Use modified JDETracker (without detection module) to track single-camera vehicles and generate single-came…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Under Display Cameras present a promising opportunity for phone manufacturers to achieve bezel-free displays by positioning the camera behind semi-transparent OLED screens. Unfortunately, such imaging systems suffer from severe image degradation due to light attenuation and diffraction effects. In this work, we present Deep Atrous Guided Filter (DAGF), a two-stage, end-to-end approach for image restoration in UDC systems. A Low-Resolution Network first restores image quality at low-resolution, which is subsequently used by the Guided Filter Network as a filtering input to produce a high-resolu…
Deep learning has achieved impressive results in camera localization, but current single-image techniques typically suffer from a lack of robustness, leading to large outliers. To some extent, this has been tackled by sequential (multi-images) or geometry constraint approaches, which can learn to reject dynamic objects and illumination conditions to achieve better performance. In this work, we show that attention can be used to force the network to focus on more geometrically robust objects and features, achieving state-of-the-art performance in common benchmark, even if using only a single im…
Enhancing Multi-Camera People Tracking with Anchor-Guided Clustering and Spatio-Temporal Consistency ID Re-Assignment
University of Washington（华盛顿大学）；Electronics and Telecommunications Research Institute（电子与电信研究所）；National Center for High-Performance Computing（国家高性能计算中心）
提出一种结合锚定引导聚类与时空一致性的多摄像机多人员跟踪方法，在CVPR AI City Challenge 2023上以95.36% IDF1获第一名。
Depth estimation is essential for various important real-world applications such as autonomous driving. However, it suffers from severe performance degradation in high-velocity scenario since traditional cameras can only capture blurred images. To deal with this problem, the spike camera is designed to capture the pixel-wise luminance intensity at high frame rate. However, depth estimation with spike camera remains very challenging using traditional monocular or stereo depth estimation algorithms, which are based on the photometric consistency. In this paper, we propose a novel Uncertainty-Gui…
Subjective Camera 1.0: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusion
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（国家多媒体软件工程研究中心，人工智能研究院，计算机科学学院，武汉大学）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（湖北省多媒体与网络通信工程重点实验室）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；CyberAgent AI Lab, Japan（CyberAgent AI实验室，日本）
本文提出Subjective Camera 1.0框架，通过文本描述和逐步草图等主观读数，利用序列感知草图引导扩散模型重建真实场景，无需大规模配对训练数据，在图像质量与空间语义对齐上达到最先进水平。
University of Pennsylvania（宾夕法尼亚大学）；Adobe Research（Adobe研究院）
Recently, deep models have established SOTA performance for low-resolution image inpainting, but they lack fidelity at resolutions associated with modern cameras such as 4K or more, and for large holes. We contribute an inpainting benchmark dataset of photos at 4K and above representative of modern sensors. We demonstrate a novel framework that combines deep learning and traditional methods. We use an existing deep inpainting model LaMa to fill the hole plausibly, establish three guide images consisting of structure, segmentation, depth, and apply a multiply-guided PatchMatch to produce eight …
ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking
Huazhong University of Science and Technology（华中科技大学）；Jiangxi University of Water Resources and Electric Power（江西水利电力大学）；Zhongnan University of Economics and Law（中南财经政法大学）
SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection
AICV Lab, University of Arkansas, USA（AICV实验室，阿肯色大学，美国）；University of Utah, USA（犹他大学，美国）；University of Houston, USA（休斯顿大学，美国）；University of Liverpool, UK（利物浦大学，英国）
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，计算机学院，北京大学）
The University of Manchester（曼彻斯特大学）；Cardiff University（卡迪夫大学）
This paper presents the development of a control system for vision-guided pick-and-place tasks using a robot arm equipped with a 3D camera. The main steps include camera intrinsic and extrinsic calibration, hand-eye calibration, initial object pose registration, objects pose alignment algorithm, and pick-and-place execution. The proposed system allows the robot be able to to pick and place object with limited times of registering a new object and the developed software can be applied for new object scenario quickly. The integrated system was tested using the hardware combination of kuka iiwa, …
Radar Guided Camera Verification for Automatic Emergency Braking Rethinking Object Detection in Radar Camera Fusion
Autonomous Systems Lab, Department of Mechatronics Engineering, SRM Institute of Science and Technology（SRM科学与技术学院机电工程系自主系统实验室）；Centre for Electric Mobility, Department of EEE, SRM Institute of Science and Technology（SRM科学与技术学院电气与电子工程系电动出行中心）
Harbin Institute of Technology（哈尔滨工业大学）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；Shenzhen University（深圳大学）
Weakly Supervised Object Localization is challenging because of the lack of bounding box annotations. Previous works tend to generate a class activation map i.e CAM to localize the object. Unfortunately, the network activates only the features that discriminate the object and does not activate the whole object. Some methods tend to remove some parts of the object to force the CNN to detect other features, whereas, others change the network structure to generate multiple CAMs from different levels of the model. In this present article, we propose to take advantage of the generalization ability …
Huazhong University of Science and Technology（华中科技大学）；Horizon Robotics（地平线机器人）
Learning Bird's Eye View (BEV) representation from surrounding-view cameras is of great importance for autonomous driving. In this work, we propose a Geometry-guided Kernel Transformer (GKT), a novel 2D-to-BEV representation learning mechanism. GKT leverages the geometric priors to guide the transformer to focus on discriminative regions and unfolds kernel features to generate BEV representation. For fast inference, we further introduce a look-up table (LUT) indexing method to get rid of the camera's calibrated parameters at runtime. GKT can run at $72.3$ FPS on 3090 GPU / $45.6$ FPS on 2080ti…
