UrbanFusion: 用于鲁棒空间表示对比学习的随机多模态融合
UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations
AI总结：
提出UrbanFusion模型，通过随机多模态融合（SMF）和Transformer模块整合街景、遥感、地图和POI数据，在56个城市41项任务中优于现有GeoAI模型。
摘要：
预测房价和公共卫生指标等城市现象需要有效整合各种地理空间数据。当前方法主要使用特定任务模型，而近期用于空间表示的通用模型通常仅支持有限模态且缺乏多模态融合能力。为克服这些挑战，我们提出UrbanFusion，一种具有随机多模态融合（SMF）的空间表示模型。该框架采用模态特定编码器处理不同类型输入，包括街景图像、遥感数据、制图地图和兴趣点（POI）数据。这些多模态输入通过基于Transformer的融合模块进行集成，学习统一表示。在全世界56个城市的41项任务上的广泛评估表明，与最先进的GeoAI模型相比，UrbanFusion具有强大的泛化和预测性能。具体而言，它1）在位置编码上优于先前模型，2）允许推理时多模态输入，3）能很好地泛化到训练中未见过的区域。UrbanFusion在预训练和推理过程中均可灵活利用给定位置的任何可用模态子集，从而在多样化的数据可用性场景中实现广泛适用性。
英文摘要：
Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data. Current methods primarily utilize task-specific models, while recent generic models for spatial representations often support only limited modalities and lack multimodal fusion capabilities. To overcome these challenges, we present UrbanFusion, a spatial representation model that features Stochastic Multimodal Fusion (SMF). The framework employs modality-specific encoders to process different types of inputs, including street view imagery, remote sensing data, cartographic maps, and points of interest (POIs) data. These multimodal inputs are integrated via a Transformer-based fusion module that learns unified representations. An extensive evaluation across 41 tasks in 56 cities worldwide demonstrates UrbanFusion's strong generalization and predictive performance compared to state-of-the-art GeoAI models. Specifically, it 1) outperforms prior models on location-encoding, 2) allows multimodal input during inference, and 3) generalizes well to regions unseen during training. UrbanFusion can flexibly utilize any subset of available modalities for a given location during both pretraining and inference, enabling broad applicability across diverse data availability scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
