arXiv 2409.078432025-11-09cs.CVcs.RO

基于师生学习与无标注数据的真实场景实时多视角全向深度估计

Real-time Multi-view Omnidirectional Depth Estimation for Real Scenarios based on Teacher-Student Learning with Unlabeled Data

School of Artificial Intelligence/School of Future Technology, Nanjing University of Information Science and Technology（人工智能学院/未来技术学院，信息科学与技术大学）；School of Electronic Science and Engineering, Nanjing University（电子科学与工程学院，南京大学）；School of Computer Engineering, Jinling Institute of Technology（计算机工程学院，金陵科技学院）；Suzhou High Technology Research Institute, Nanjing University（苏州高新技术研究院，南京大学）

Ming Li, Xiong Yang, Chaofan Wu, Jiaheng Li, Pinzhi Wang, Xuejiao Hu, Sidan Du, Yang Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出实时全向深度估计方法Rt-OmniMVS及HexaMODE系统，通过师生学习利用无标注真实数据训练轻量模型，在边缘平台以15 FPS实现高精度、强泛化的360度深度感知。

摘要：

全向深度估计能够在完整的360度范围内实现高效的三维感知。然而，在自动驾驶和机器人等实际应用中，现有算法在实现实时性能和跨场景鲁棒泛化方面仍面临重大挑战。本文提出了一种面向边缘计算平台的实时全向深度估计方法，名为Rt-OmniMVS，该方法引入了组合球面扫描方法，并实现了轻量级网络结构，从而在边缘计算平台上实现实时性能。为了在真实环境中获得高精度、鲁棒性和泛化能力，我们引入了一种师生学习策略。我们利用高精度立体匹配方法作为教师模型，为无标注的真实世界数据预测伪标签，并利用数据和模型增强技术进行训练，以提升学生模型Rt-OmniMVS的性能。我们还提出了HexaMODE，一种基于多视角鱼眼相机和边缘计算设备的全向深度感知系统。我们采集了一个包含无标注真实世界数据和合成数据的大规模混合数据集用于模型训练。在公开数据集上的实验表明，所提出的方法在显著降低资源消耗的同时，取得了与最先进方法相当的结果。所提出的系统和算法在室内外各种复杂真实场景中也表现出高精度，在边缘计算平台上实现了每秒15帧的推理速度。

英文摘要：

Omnidirectional depth estimation enables efficient 3D perception over a full 360-degree range. However, in real-world applications such as autonomous driving and robotics, achieving real-time performance and robust cross-scene generalization remains a significant challenge for existing algorithms. In this paper, we propose a real-time omnidirectional depth estimation method for edge computing platforms named Rt-OmniMVS, which introduces the Combined Spherical Sweeping method and implements the lightweight network structure to achieve real-time performance on edge computing platforms. To achieve high accuracy, robustness, and generalization in real-world environments, we introduce a teacher-student learning strategy. We leverage the high-precision stereo matching method as the teacher model to predict pseudo labels for unlabeled real-world data, and utilize data and model augmentation techniques for training to enhance performance of the student model Rt-OmniMVS. We also propose HexaMODE, an omnidirectional depth sensing system based on multi-view fisheye cameras and edge computation device. A large-scale hybrid dataset contains both unlabeled real-world data and synthetic data is collected for model training. Experiments on public datasets demonstrate that proposed method achieves results comparable to state-of-the-art approaches while consuming significantly less resource. The proposed system and algorithm also demonstrate high accuracy in various complex real-world scenarios, both indoors and outdoors, achieving an inference speed of 15 frames per second on edge computing platforms.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。