ThinkGeo: 评估用于遥感任务的工具增强代理
ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks
AI总结：
ThinkGeo是一个针对遥感任务设计的代理基准，评估LLM驱动代理在结构化工具使用和多步骤规划中的能力，通过人类 curated 查询和专家验证的推理步骤测试不同模型的工具准确性和规划一致性。
摘要：
近年来，大型语言模型（LLMs）的进步使得能够通过逐步推理解决复杂现实任务的工具增强代理成为可能。然而，现有评估往往集中在通用或多模态场景，缺乏对复杂遥感用例中工具使用能力的领域特定基准。我们提出了ThinkGeo，一个代理基准，旨在通过结构化工具使用和多步骤规划评估LLM驱动的代理在遥感任务中的表现。受工具交互范式启发，ThinkGeo包含人类curated的查询，涵盖城市规划、灾害评估和变化分析、环境监测、交通分析、航空监控、休闲基础设施和工业站点分析等广泛的实际应用。查询基于卫星或航空影像，包括光学RGB和SAR数据，并要求代理使用多样化的工具集进行推理。我们实现了一种ReAct风格的交互循环，并在486个结构化代理任务上评估了开源和闭源LLM（如GPT-4o、Qwen2.5），共包含1,778个专家验证的推理步骤。该基准报告了逐步执行指标和最终答案正确性。我们的分析揭示了不同模型在工具准确性和规划一致性方面的显著差异。ThinkGeo为评估工具增强LLM如何处理遥感中的空间推理提供了首个广泛的测试平台。
英文摘要：
Recent progress in large language models (LLMs) has enabled tool-augmented agents capable of solving complex real-world tasks through step-by-step reasoning. However, existing evaluations often focus on general-purpose or multimodal scenarios, leaving a gap in domain-specific benchmarks that assess tool-use capabilities in complex remote sensing use cases. We present ThinkGeo, an agentic benchmark designed to evaluate LLM-driven agents on remote sensing tasks via structured tool use and multi-step planning. Inspired by tool-interaction paradigms, ThinkGeo includes human-curated queries spanning a wide range of real-world applications such as urban planning, disaster assessment and change analysis, environmental monitoring, transportation analysis, aviation monitoring, recreational infrastructure, and industrial site analysis. Queries are grounded in satellite or aerial imagery, including both optical RGB and SAR data, and require agents to reason through a diverse toolset. We implement a ReAct-style interaction loop and evaluate both open and closed-source LLMs (e.g., GPT-4o, Qwen2.5) on 486 structured agentic tasks with 1,778 expert-verified reasoning steps. The benchmark reports both step-wise execution metrics and final answer correctness. Our analysis reveals notable disparities in tool accuracy and planning consistency across models. ThinkGeo provides the first extensive testbed for evaluating how tool-enabled LLMs handle spatial reasoning in remote sensing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
