arXiv 2602.083422026-02-09cs.CVcs.AI

UrbanGraphEmbeddings: 学习和评估空间导向的多模态嵌入用于城市科学

UrbanGraphEmbeddings: Learning and Evaluating Spatially Grounded Multimodal Embeddings for Urban Science

National University of Singapore Department of Architecture Singapore；The Chinese University of Hong Kong Dept of Systems Eng. \& Eng. Mgmt. China；Singapore Management University School of Computing \& Info. Systems Singapore；National University of Singapore；Department of Architecture；The Chinese University of Hong Kong；Singapore Management University

Jie Zhang, Xingtong Yu, Yuan Fang, Rudi Stouffs, Zdravko Trivic

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出UGE框架，通过空间导向的多模态嵌入提升城市理解任务性能，实验显示在图像检索和地理位置排名上取得显著提升。

摘要：

学习适用于城市环境的可转移多模态嵌入具有挑战性，因为城市理解本质上是空间性的，但现有数据集和基准缺乏街道视图图像与城市结构之间的显式对齐。我们介绍了UGData，一个空间导向的数据集，将街道视图图像锚定到结构化的空间图上，并通过空间推理路径和空间上下文描述提供图对齐的监督，揭示距离、方向性、连接性和邻里上下文，超越图像内容。基于UGData，我们提出UGE，一种两阶段训练策略，通过结合指令引导的对比学习和基于图的空间编码，逐步且稳定地对齐图像、文本和空间结构。我们最后引入UGBench，一个综合的基准，用于评估空间导向嵌入如何支持多样化的城市理解任务——包括地理位置排名、图像检索、城市感知和空间导向。我们开发了UGE在多个最先进的VLM基础模型上，包括Qwen2-VL、Qwen2.5-VL、Phi-3-Vision和LLaVA1.6-Mistral，并通过LoRA微调训练固定维度的空间嵌入。基于Qwen2.5-VL-7B基础模型的UGE在训练城市上实现了图像检索高达44%的提升和地理位置排名30%的提升，在外推城市上分别实现了超过30%和22%的提升，证明了显式空间导向对空间密集城市任务的有效性。

英文摘要：

Learning transferable multimodal embeddings for urban environments is challenging because urban understanding is inherently spatial, yet existing datasets and benchmarks lack explicit alignment between street-view images and urban structure. We introduce UGData, a spatially grounded dataset that anchors street-view images to structured spatial graphs and provides graph-aligned supervision via spatial reasoning paths and spatial context captions, exposing distance, directionality, connectivity, and neighborhood context beyond image content. Building on UGData, we propose UGE, a two-stage training strategy that progressively and stably aligns images, text, and spatial structures by combining instruction-guided contrastive learning with graph-based spatial encoding. We finally introduce UGBench, a comprehensive benchmark to evaluate how spatially grounded embeddings support diverse urban understanding tasks -- including geolocation ranking, image retrieval, urban perception, and spatial grounding. We develop UGE on multiple state-of-the-art VLM backbones, including Qwen2-VL, Qwen2.5-VL, Phi-3-Vision, and LLaVA1.6-Mistral, and train fixed-dimensional spatial embeddings with LoRA tuning. UGE built upon Qwen2.5-VL-7B backbone achieves up to 44% improvement in image retrieval and 30% in geolocation ranking on training cities, and over 30% and 22% gains respectively on held-out cities, demonstrating the effectiveness of explicit spatial grounding for spatially intensive urban tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。