arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.01334cs.CV
收藏

STEPS:联合自监督夜间图像增强和深度估计

STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Xi’an Research Institute of High-Tech(西安高新技术研究所);Intel Labs China(英特尔中国研究院);Peking University(北京大学)

Yupeng Zheng, Chengliang Zhong, Pengfei Li, Huan-ang Gao, Yuhang Zheng, Bu Jin, Ling Wang, Hao Zhao, Guyue Zhou, Qichao Zhang, Dongbin Zhao

摘要:

自监督的深度估计最近引起了很多关注,因为它可以提高自动驾驶车辆的3D感知能力。然而,它本质上依赖于光度一致性假设,这在夜间几乎不成立。虽然已经提出了各种监督的夜间图像增强方法,它们在具有挑战性的驾驶场景中的泛化性能并不令人满意。为此,我们提出了第一种方法,该方法联合学习夜间图像增强器和深度估计器,而不使用地面实况。我们的方法紧密纠缠两个自我监督的任务,使用新提出的不确定像素掩蔽策略。这种策略起源于观察,夜间图像不仅遭受曝光不足的地区,但也从曝光过度的地区。通过将桥形曲线拟合到照明图分布,两个区域都被抑制,两个任务自然地桥接。我们在两个已建立的数据集上对该方法进行了基准测试:nuScenes和RobotCar,并在这两个数据集上展示了最先进的性能。详细的消融也揭示了我们的建议的机制。最后但并非最不重要的是,为了缓解现有数据集的稀疏地面实况的问题,我们提供了一个新的基于CARLA的照片逼真增强的夜间数据集。这给社会带来了新的挑战。代码、数据和模型可在https://github.com/ucaszyp/STEPS上获得。

英文摘要:

Self-supervised depth estimation draws a lot of attention recently as it can promote the 3D sensing capabilities of self-driving vehicles. However, it intrinsically relies upon the photometric consistency assumption, which hardly holds during nighttime. Although various supervised nighttime image enhancement methods have been proposed, their generalization performance in challenging driving scenarios is not satisfactory. To this end, we propose the first method that jointly learns a nighttime image enhancer and a depth estimator, without using ground truth for either task. Our method tightly entangles two self-supervised tasks using a newly proposed uncertain pixel masking strategy. This strategy originates from the observation that nighttime images not only suffer from underexposed regions but also from overexposed regions. By fitting a bridge-shaped curve to the illumination map distribution, both regions are suppressed and two tasks are bridged naturally. We benchmark the method on two established datasets: nuScenes and RobotCar and demonstrate state-of-the-art performance on both of them. Detailed ablations also reveal the mechanism of our proposal. Last but not least, to mitigate the problem of sparse ground truth of existing datasets, we provide a new photo-realistically enhanced nighttime dataset based upon CARLA. It brings meaningful new challenges to the community. Codes, data, and models are available at https://github.com/ucaszyp/STEPS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑