In this paper, we formulate a potentially valuable panoramic depth completion (PDC) task as panoramic 3D cameras often produce 360° depth with missing data in complex scenes. Its goal is to recover dense panoramic depths from raw sparse ones and panoramic RGB images. To deal with the PDC task, we train a deep network that takes both depth and image as inputs for the dense panoramic depth recovery. However, it needs to face a challenging optimization problem of the network parameters due to its non-convex objective function. To address this problem, we propose a simple yet effective approach termed M{^3}PT: multi-modal masked pre-training. Specifically, during pre-training, we simultaneously cover up patches of the panoramic RGB image and sparse depth by shared random mask, then reconstruct the sparse depth in the masked regions. To our best knowledge, it is the first time that we show the effectiveness of masked pre-training in a multi-modal vision task, instead of the single-modal task resolved by masked autoencoders (MAE). Different from MAE where fine-tuning completely discards the decoder part of pre-training, there is no architectural difference between the pre-training and fine-tuning stages in our M$^{3}$PT as they only differ in the prediction density, which potentially makes the transfer learning more convenient and effective. Extensive experiments verify the effectiveness of M{^3}PT on three panoramic datasets. Notably, we improve the state-of-the-art baselines by averagely 26.2% in RMSE, 51.7% in MRE, 49.7% in MAE, and 37.5% in RMSElog on three benchmark datasets.
Recent automotive vision work has focused almost exclusively on processing forward-facing cameras. However, future autonomous vehicles will not be viable without a more comprehensive surround sensing, akin to a human driver, as can be provided by 360° panoramic cameras. We present an approach to adapt contemporary deep network architectures developed on conventional rectilinear imagery to work on equirectangular 360° panoramic imagery. To address the lack of annotated panoramic automotive datasets availability, we adapt a contemporary automotive dataset, via style and projection transformation…
The computer-assisted radiologic informative report is currently emerging in dental practice to facilitate dental care and reduce time consumption in manual panoramic radiographic interpretation. However, the amount of dental radiographs for training is very limited, particularly from the point of view of deep learning. This study aims to utilize recent self-supervised learning methods like SimMIM and UM-MAE to increase the model efficiency and understanding of the limited number of dental radiographs. We use the Swin Transformer for teeth numbering, detection of dental restorations, and insta…
Conditional Panoramic Image Generation via Masked Autoregressive Modeling
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Insta360 Research（Insta360研究院）；National University of Singapore（新加坡国立大学）；The University of Tokyo（东京大学）
MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders
S-Lab, Nanyang Technological University（新加坡南洋理工大学S-Lab）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；University of Chinese Academy of Sciences（中国科学院大学）
Korea University（高丽大学）；University of Bologna（博洛尼亚大学）
We propose MaskingDepth, a novel semi-supervised learning framework for monocular depth estimation to mitigate the reliance on large ground-truth depth quantities. MaskingDepth is designed to enforce consistency between the strongly-augmented unlabeled data and the pseudo-labels derived from weakly-augmented unlabeled data, which enables learning depth without supervision. In this framework, a novel data augmentation is proposed to take the advantage of a naive masking strategy as an augmentation, while avoiding its scale ambiguity problem between depths from weakly- and strongly-augmented bra…
Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Intelligent Computing Technology, CAS（中国科学院智能计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）
360ORB-SLAM: A Visual SLAM System for Panoramic Images with Depth Completion Network
Hangzhou Normal University（杭州师范大学）；Chinese Academy of Sciences（中国科学院）；Quanzhou Institute of Equipment Manufacturing（泉州装备制造研究所）；Hangzhou Dianzi University（杭州电子科技大学）；Tianjin University of Technology（天津理工大学）
School of Mechanical and Robotics Engineering, Gwangju Institute of Science and Technology (GIST)（光州科学技术学院机械与机器人工程学院）；Department of Smart ICT Convergence Engineering, Seoul National University of Science and Technology（首尔科学技术大学智能ICT融合工程系）
In this paper, we propose a learning-based method for predicting dense depth values of a scene from a monocular omnidirectional image. An omnidirectional image has a full field-of-view, providing much more complete descriptions of the scene than perspective images. However, fully-convolutional networks that most current solutions rely on fail to capture rich global contexts from the panorama. To address this issue and also the distortion of equirectangular projection in the panorama, we propose Cubemap Vision Transformers (CViT), a new transformer-based architecture that can model long-range d…
Geometry and Gradient-based Partitioning for Panoramic Outdoor Reconstruction
Insta360 Research（Insta360研究机构）；Sun Yat-sen University（中山大学）；South China University of Technology（华南理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Engineering University（哈尔滨工程大学）；Wuhan University（武汉大学）
We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issu…
University of Chinese Academy of Sciences（中国科学院大学）；Chongqing University（重庆大学）；KAUST（阿卜杜拉国王科技大学）
Depth estimation is a crucial step for 3D reconstruction with panorama images in recent years. Panorama images maintain the complete spatial information but introduce distortion with equirectangular projection. In this paper, we propose an ACDNet based on the adaptively combined dilated convolution to predict the dense depth map for a monocular panoramic image. Specifically, we combine the convolution kernels with different dilations to extend the receptive field in the equirectangular projection. Meanwhile, we introduce an adaptive channel-wise fusion module to summarize the feature maps and …
Zhejiang University（浙江大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；National Engineering Research Center of Optical Instrumentation, Zhejiang University（浙江大学国家光学仪器工程技术研究中心）
Depth estimation, as a necessary clue to convert 2D images into the 3D space, has been applied in many machine vision areas. However, to achieve an entire surrounding 360-degree geometric sensing, traditional stereo matching algorithms for depth estimation are limited due to large noise, low accuracy, and strict requirements for multi-camera calibration. In this work, for a unified surrounding perception, we introduce panoramic images to obtain larger field of view. We extend PADENet first appeared in our previous conference work for outdoor scene understanding, to perform panoramic monocular …
Naval Research Laboratory（海军研究实验室）；California Polytechnic State University, San Luis Obispo（加州州立理工大学圣路易斯奥比斯波分校）
We introduce a convolutional neural network model for unsupervised learning of depth and ego-motion from cylindrical panoramic video. Panoramic depth estimation is an important technology for applications such as virtual reality, 3D modeling, and autonomous robotic navigation. In contrast to previous approaches for applying convolutional neural networks to panoramic imagery, we use the cylindrical panoramic projection which allows for the use of the traditional CNN layers such as convolutional filters and max pooling without modification. Our evaluation of synthetic and real data shows that un…
Beijing Jiaotong University（北京交通大学）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）
For a monocular 360 image, depth estimation is a challenging because the distortion increases along the latitude. To perceive the distortion, existing methods devote to designing a deep and complex network architecture. In this paper, we provide a new perspective that constructs an interpretable and sparse representation for a 360 image. Considering the importance of the geometric structure in depth estimation, we utilize the contourlet transform to capture an explicit geometric cue in the spectral domain and integrate it with an implicit cue in the spatial domain. Specifically, we propose a n…
NavInfo Europe（四维图新欧洲公司）；Eindhoven University of Technology（埃因霍温理工大学）
Self-supervised monocular depth estimation is a salient task for 3D scene understanding. Learned jointly with monocular ego-motion estimation, several methods have been proposed to predict accurate pixel-wise depth without using labeled data. Nevertheless, these methods focus on improving performance under ideal conditions without natural or digital corruptions. The general absence of occlusions is assumed even for object-specific depth estimation. These methods are also vulnerable to adversarial attacks, which is a pertinent concern for their reliable deployment in robots and autonomous drivi…
East China University of Science and Technology（华东理工大学）；Oklahoma State University（俄克拉荷马州立大学）
Previous work has shown that adversarial learning can be used for unsupervised monocular depth and visual odometry (VO) estimation, in which the adversarial loss and the geometric image reconstruction loss are utilized as the mainly supervisory signals to train the whole unsupervised framework. However, the performance of the adversarial framework and image reconstruction is usually limited by occlusions and the visual field changes between frames. This paper proposes a masked generative adversarial network (GAN) for unsupervised monocular depth and ego-motion estimation.The MaskNet and Boolea…
Monocular depth estimation has been extensively explored based on deep learning, yet its accuracy and generalization ability still lag far behind the stereo-based methods. To tackle this, a few recent studies have proposed to supervise the monocular depth estimation network by distilling disparity maps as proxy ground-truths. However, these studies naively distill the stereo knowledge without considering the comparative advantages of stereo-based and monocular depth estimation methods. In this paper, we propose to selectively distill the disparity maps for more reliable proxy supervision. Spec…
MaskHOI: Robust 3D Hand-Object Interaction Estimation via Masked Pre-training
PCA Lab, School of Intelligence Science and Technology, Nanjing University（PCA实验室，智能科学与技术学院，南京大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Science and Technology（南京理工大学）；Nankai University（南开大学）
Semi-supervised classification of dental conditions in panoramic radiographs using large language model and instance segmentation: A real-world dataset evaluation
Institute of Computing, Federal University of Bahia（巴伊亚联邦大学计算学院）；Federal Institute of Bahia（巴伊亚联邦学院）；Federal University of Maranhão（马拉尼昂联邦大学）；Harvard School of Public Health（哈佛大学公共卫生学院）；Instituto de Ciência e Inovação em Engenharia Mecânica e Engenharia Industrial（机械与工业工程科学与创新研究所）；Faculdade de Engenharia, Universidade do Porto（波尔图大学工程学院）；Dentistry Faculty, Federal University of Bahia（巴伊亚联邦大学牙科学院）
In this paper, we consider adversarial attacks against a system of monocular depth estimation (MDE) based on convolutional neural networks (CNNs). The motivation is two-fold. One is to study the security of MDE systems, which has not been actively considered in the community. The other is to improve our understanding of the computational mechanism of CNNs performing MDE. Toward this end, we apply the method recently proposed for visualization of MDE to defending attacks. It trains another CNN to predict a saliency map from an input image, such that the CNN for MDE continues to accurately estim…
MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning
School of EIC, Huazhong University of Science & Technology（华中科技大学电子信息与通信学院）；Institute of Artificial Intelligence, Huazhong University of Science & Technology（华中科技大学人工智能研究院）；Horizon Robotics
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure AI（微软Azure人工智能）
3D photography renders a static image into a video with appealing 3D visual effects. Existing approaches typically first conduct monocular depth estimation, then render the input frame to subsequent frames with various viewpoints, and finally use an inpainting model to fill those missing/occluded regions. The inpainting model plays a crucial role in rendering quality, but it is normally trained on out-of-domain data. To reduce the training and inference gap, we propose a novel self-supervised diffusion model as the inpainting module. Given a single input image, we automatically construct a tra…
Masked Image Modeling (MIM) has recently been established as a potent pre-training paradigm. A pretext task is constructed by masking patches in an input image, and this masked content is then predicted by a neural network using visible patches as sole input. This pre-training leads to state-of-the-art performance when finetuned for high-level semantic tasks, e.g. image classification and object detection. In this paper we instead seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks, such as depth prediction or optical flow …
Revisiting Cross-View Completion: Self-Supervised Pre-Training via Reconstruction Error Comparison
LIGM, Ecole Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS（LIGM实验室、巴黎国立路桥学校、巴黎IP大学、古斯塔夫·埃菲尔大学、法国国家科学研究中心）；Univ. Bordeaux, CNRS, Bordeaux INP, IMS（波尔多大学、法国国家科学研究中心、波尔多国立综合理工学院、IMS研究所）
From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance
Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院）；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院）；Kyungpook National University（庆北大学）
