太阳落山，灯光亮起：面向鲁棒语义感知的真实感单目夜间模拟
Sun Off, Lights On: Photorealistic Monocular Nighttime Simulation for Robust Semantic Perception
AI总结：
本文提出SOLO方法，首次在单张白天图像上通过三维几何、材质与光源估计及语义感知的光源概率实例化和光线追踪，生成真实感夜间图像，并在昼夜自适应的夜间语义分割中验证其有效性。
摘要：
夜间场景对于学习模型而言难以进行语义感知，对于人类而言也难以标注。因此，逼真的合成夜间数据对于学习夜间鲁棒语义感知变得更加重要，因为它们具有准确且成本低廉的语义标注。然而，现有的从白天对应图像生成夜间图像的数据驱动或手工设计技术在真实感方面表现不佳。原因在于高度空间变化的夜间光照与其白天对应光照差异巨大，并与场景中具有空间变化材质的物体发生复杂交互，这种交互发生在三维空间中，很难用此类二维方法捕捉。上述三维交互和光照变化在文献中已被证明与其他条件（如雾或雨）相比同样难以建模。我们的方法名为 Sun Off, Lights On（SOLO），是首个通过在三维空间中操作、以真实感方式对单张图像进行夜间模拟的方法。该方法首先从输入的白天图像中显式估计场景的三维几何、材质和光源位置，并通过以概率方式实例化光源来重新照亮场景，该方式考虑了光源的语义信息，然后运行标准光线追踪。我们的夜间图像不仅在视觉质量和真实感上优于包括扩散模型在内的竞争方法，而且这些图像还被证明在昼夜自适应中的夜间语义分割方面更有益处。代码和数据将公开发布。
英文摘要：
Nighttime scenes are hard to semantically perceive with learned models and annotate for humans. Thus, realistic synthetic nighttime data become all the more important for learning robust semantic perception at night, thanks to their accurate and cheap semantic annotations. However, existing data-driven or hand-crafted techniques for generating nighttime images from daytime counterparts suffer from poor realism. The reason is the complex interaction of highly spatially varying nighttime illumination, which differs drastically from its daytime counterpart, with objects of spatially varying materials in the scene, happening in 3D and being very hard to capture with such 2D approaches. The above 3D interaction and illumination shift have proven equally hard to model in the literature, as opposed to other conditions such as fog or rain. Our method, named Sun Off, Lights On (SOLO), is the first to perform nighttime simulation on single images in a photorealistic fashion by operating in 3D. It first explicitly estimates the 3D geometry, the materials and the locations of light sources of the scene from the input daytime image and relights the scene by probabilistically instantiating light sources in a way that accounts for their semantics and then running standard ray tracing. Not only is the visual quality and photorealism of our nighttime images superior to competing approaches including diffusion models, but the former images are also proven more beneficial for semantic nighttime segmentation in day-to-night adaptation. Code and data will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
