GeoAgentBench: 一种面向空间分析工具增强代理的动态执行基准
GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis
AI总结：
本文提出GeoAgentBench，通过动态执行沙盒评估地理信息代理，引入PEA指标和视觉语言模型验证，改进Plan-and-React架构，提升空间分析代理的执行鲁棒性与逻辑严谨性。
摘要：
大型语言模型（LLMs）整合到地理信息系统（GIS）中标志着向自主空间分析的范式转变。然而，评估这些LLM代理仍具挑战性，因地理空间工作流程复杂且多步骤。现有基准主要依赖静态文本或代码匹配，忽视动态运行时反馈和空间输出的多模态性质。为此，我们引入GeoAgentBench（GABench），一种动态交互性评估基准，专为工具增强GIS代理设计。GABench提供一个现实执行沙盒，整合117个原子GIS工具，涵盖6个核心GIS领域中的53种典型空间分析任务。我们设计了参数执行精度（PEA）指标，利用“最后一次尝试对齐”策略量化隐式参数推断的保真度。此外，提出基于视觉语言模型（VLM）的验证方法，评估数据-空间准确性和制图风格的符合性。为进一步解决频繁的任务失败问题，我们开发了新的代理架构Plan-and-React，通过解耦全局协调与分步反应执行，模仿专家认知流程。对七种代表性LLM的大量实验表明，Plan-and-React范式显著优于传统框架，在多步骤推理和错误恢复方面取得最佳平衡。我们的发现突显了当前能力边界，并建立了评估和推进下一代自主GeoAI的稳健标准。
英文摘要：
The integration of Large Language Models (LLMs) into Geographic Information Systems (GIS) marks a paradigm shift toward autonomous spatial analysis. However, evaluating these LLM-based agents remains challenging due to the complex, multi-step nature of geospatial workflows. Existing benchmarks primarily rely on static text or code matching, neglecting dynamic runtime feedback and the multimodal nature of spatial outputs. To address this gap, we introduce GeoAgentBench (GABench), a dynamic and interactive evaluation benchmark tailored for tool-augmented GIS agents. GABench provides a realistic execution sandbox integrating 117 atomic GIS tools, encompassing 53 typical spatial analysis tasks across 6 core GIS domains. Recognizing that precise parameter configuration is the primary determinant of execution success in dynamic GIS environments, we designed the Parameter Execution Accuracy (PEA) metric, which utilizes a "Last-Attempt Alignment" strategy to quantify the fidelity of implicit parameter inference. Complementing this, a Vision-Language Model (VLM) based verification is proposed to assess data-spatial accuracy and cartographic style adherence. Furthermore, to address the frequent task failures caused by parameter misalignments and runtime anomalies, we developed a novel agent architecture, Plan-and-React, that mimics expert cognitive workflows by decoupling global orchestration from step-wise reactive execution. Extensive experiments with seven representative LLMs demonstrate that the Plan-and-React paradigm significantly outperforms traditional frameworks, achieving the optimal balance between logical rigor and execution robustness, particularly in multi-step reasoning and error recovery. Our findings highlight current capability boundaries and establish a robust standard for assessing and advancing the next generation of autonomous GeoAI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
