多模态大模型模态迁移：自主地理信息系统智能体的先决条件
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
浏览论文内容
中文总结 AI 辅助
本文针对自主GIS智能体的先决条件，提出LMM模态迁移任务，发现现有LMM在图像与文本模态间传递空间信息的能力不足，需加强多模态对齐以实现地理空间理解。
中文摘要 AI 辅助
AI模型在理解和处理空间信息方面的能力日益提升，这推动了智能体在空间任务与工作流中的问题解决能力。然而，多数关于其空间能力（如空间推理）的研究聚焦于文本模态作为输入与输出，这与人类的地理信息系统（GIS）工作流形成对比——人类在GIS工作流中常结合、交替使用文本与视觉模态，二者互为补充。因此，要真正实现自动化GIS分析流水线或执行人类设计的GIS工作流，AI模型，尤其是多模态大模型（Large Multimodal Models, LMMs），需能在这类工作流传统使用的图像与文本模态间无缝转换。本文提出一项模态迁移任务：（1）要求LMM先描述输入的规则网格彩色方块图像；（2）要求新的LMM实例利用前者输出的文本描述，重新生成原始空间场景的图像。该任务可量化LMM在图像与文本模态间传递空间信息的能力。最终，通过从空间信息理论视角考察LMM的模态迁移能力，本研究揭示了一个关键瓶颈：要让LMM实现强大且鲁棒的地理空间理解，需进行严格的多模态对齐。我们的结果表明，近期的LMM（此处来自OpenAI）在被要求重新生成简单的彩色方块空间网格图像时，仍在模态迁移方面存在困难。
英文摘要
AI models are becoming increasingly adept at understanding and processing spatial information, thereby facilitating agentic problem-solving in spatial tasks and workflows. However, most of the research on their spatial capabilities (e.g., spatial reasoning) has focused on the textual modality as input and output. This contrasts with the human approach to GIS workflows, where text and visual modalities are often used together, interchangeably, and in a complementary manner. Thus, to truly achieve an automated GIS analysis pipeline or carry out human-designed GIS workflows, AI models --- Large Multimodal Models (LMMs) in particular --- need to be able to seamlessly transition between image- and text-based modalities that are traditionally used in such workflows. We present a modality transfer task that (1) asks an LMM to first describe an input image of colored squares in a regular grid, and (2) asks a new LMM instance to re-generate an image of the original spatial scene using the textual description output by the former model. This task quantifies the ability of LMMs to transfer spatial information between image and text modalities. Ultimately, by examining the modality transfer capability of LMMs through the lens of spatial information theory, this work highlights a critical bottleneck: achieving strong and robust geospatial understanding in LMMs requires rigorous, multi-modal alignment. Our results indicate that recent LMMs (here from OpenAI) still struggle with modality transfer, when tasked with re-generating an image of a simple spatial grid of color squares.
发表机构
- Graz University of Technology（格拉茨工业大学）
- University of Vienna（维也纳大学）
- University of Liverpool（利物浦大学）
机构由 AI 辅助整理，请以论文原文为准。
