We present a novel multi-view training framework and CNN architecture for combining information from multiple overlapping satellite images and noisy training labels derived from OpenStreetMap (OSM) to semantically label buildings and roads across large geographic regions (100 km$^2$). Our approach to multi-view semantic segmentation yields a 4-7% improvement in the per-class IoU scores compared to the traditional approaches that use the views independently of one another. A unique (and, perhaps, surprising) property of our system is that modifications that are added to the tail-end of the CNN for learning from the multi-view data can be discarded at the time of inference with a relatively small penalty in the overall performance. This implies that the benefits of training using multiple views are absorbed by all the layers of the network. Additionally, our approach only adds a small overhead in terms of the GPU-memory consumption even when training with as many as 32 views per scene. The system we present is end-to-end automated, which facilitates comparing the classifiers trained directly on true orthophotos vis-a-vis first training them on the off-nadir images and subsequently translating the predicted labels to geographical coordinates. With no human supervision, our IoU scores for the buildings and roads classes are 0.8 and 0.64 respectively which are better than state-of-the-art approaches that use OSM labels and that are not completely automated.
OSMLoc: Single Image-Based Visual Localization in OpenStreetMap with Fused Geometric and Semantic Guidance
Wuhan University（武汉大学）；Technical University of Munich（慕尼黑工业大学）；Nanyang Technological University（南洋理工大学）；Norwegian University of Science and Technology（挪威科学技术大学）
GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data
School of Earth and Space Sciences, Peking University（地球与空间科学学院，北京大学）；College of Urban and Environmental Sciences, Peking University（城市与环境科学学院，北京大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation, CAS（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Intelligent Maintenance and Operations Systems Lab（智能维护与运营系统实验室）
全景图像分割是一项计算机视觉任务，即在图像中找到像素组并为其分配语义类和对象实例标识符。由于图像分割在机器人和自动驾驶中的重要应用，图像分割的研究变得越来越受欢迎。因此，研究界依赖于公开可用的基准数据集来推进计算机视觉的最新发展。然而，由于密集标记图像的高成本，缺乏适用于全景分割的公开可用的地面实况标记。高标签成本也使得将现有数据集扩展到视频域和多摄像机设置具有挑战性。因此，我们提出了Waymo开放数据集：全景视频全景分割数据集，这是一个大规模的数据集，为自动驾驶提供高质量的全景分割标签。我们使用公开的Waymo Open Dataset生成数据集，利用各种相机图像。我们的标签在视频处理过程中始终保持一致，并且在车辆上安装的多个摄像头之间保持一致，以实现完整的全景场景理解。具体来说，我们为28个语义类别和2，860个时间序列提供了标签，这些序列由安装在三个不同地理位置的自动驾驶汽车上的五个摄像头捕获，总共产生了10万个标记的摄像头图像。据我们所知，这使得我们的数据集比提供视频全景分割标签的现有数据集大一个数量级。我们进一步提出了全景视频全景分割的新基准，并基于DeepLab系列模型建立了一些强基线。我们将公开基准测试和代码。在https://waymo.com/open上找到数据集。
